2024-03-16T10:44:07.564 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T10:44:07.564 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} --archive /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383 --verbose --name pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/orig.config.yaml 2024-03-16T10:44:07.615 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T10:44:07.703 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} email: pdonnell@redhat.com first_in_suite: false job_id: '7602383' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLnzJV8ne5gyMIf4urpZbqqOlFYqmyuQrZ84m/vYtwhJEPOOCnWA0r4FQLlgUfWqimFrPtiEAvgk8oTorFKu604= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCada2N07IWP7+LB714V7XpZGmIcV/4JdlYfwZDuV2LVJZwQnEpnN6kIlHzodmn8slSz0nSuW9XfkIz2zd+scz0= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLa/MvBntgRDRa3uM9RWScCoK7JGGM2OEGQQFG6Q3S8KBqu94BYRpJjQp5T9A3+8mevxB8ToVb7veo2yhthHagY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsync-tester.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_13:00:41 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T10:44:07.704 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa; will attempt to use it 2024-03-16T10:44:07.704 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks 2024-03-16T10:44:07.704 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T10:44:07.706 INFO:teuthology.task.internal:Checking packages... 2024-03-16T10:44:07.728 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-16T10:44:07.728 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T10:44:07.728 INFO:teuthology.packaging:ref: None 2024-03-16T10:44:07.728 INFO:teuthology.packaging:tag: None 2024-03-16T10:44:07.728 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-16T10:44:07.728 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:44:07.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-16T10:44:07.865 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-16T10:44:07.867 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T10:44:07.873 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T10:44:07.873 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T10:44:07.888 INFO:teuthology.task.internal:Saving configuration 2024-03-16T10:44:07.906 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T10:44:07.911 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T10:44:07.935 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 10:36:34.462209', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLnzJV8ne5gyMIf4urpZbqqOlFYqmyuQrZ84m/vYtwhJEPOOCnWA0r4FQLlgUfWqimFrPtiEAvgk8oTorFKu604='} 2024-03-16T10:44:07.959 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 10:36:34.458931', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCada2N07IWP7+LB714V7XpZGmIcV/4JdlYfwZDuV2LVJZwQnEpnN6kIlHzodmn8slSz0nSuW9XfkIz2zd+scz0='} 2024-03-16T10:44:08.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 10:36:34.460556', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLa/MvBntgRDRa3uM9RWScCoK7JGGM2OEGQQFG6Q3S8KBqu94BYRpJjQp5T9A3+8mevxB8ToVb7veo2yhthHagY='} 2024-03-16T10:44:08.315 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T10:44:08.322 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-16T10:44:08.322 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-16T10:44:08.323 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-16T10:44:08.323 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T10:44:08.557 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5966830160>, signals=[15]) 2024-03-16T10:44:08.558 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T10:44:08.564 INFO:teuthology.task.internal:Opening connections... 2024-03-16T10:44:08.564 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T10:44:08.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:44:08.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T10:44:08.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:44:08.748 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-16T10:44:08.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:44:08.835 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T10:44:08.842 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-03-16T10:44:08.860 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-03-16T10:44:08.860 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-03-16T10:44:08.917 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-03-16T10:44:08.917 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-03-16T10:44:08.917 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-03-16T10:44:08.917 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-03-16T10:44:08.917 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T10:44:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T10:44:08.919 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-03-16T10:44:08.939 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-16T10:44:08.957 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-16T10:44:08.957 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-16T10:44:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-16T10:44:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T10:44:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T10:44:09.016 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-16T10:44:09.039 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-16T10:44:09.057 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-16T10:44:09.057 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-16T10:44:09.114 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T10:44:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T10:44:09.116 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-16T10:44:24.695 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T10:44:24.706 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T10:44:24.710 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T10:44:24.711 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-03-16T10:44:24.714 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-16T10:44:24.716 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-16T10:44:24.733 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T10:44:24.739 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T10:44:24.739 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-03-16T10:44:24.772 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-16T10:44:24.774 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-16T10:44:24.872 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T10:44:24.953 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T10:44:24.969 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-16T10:44:24.969 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-16T10:44:24.970 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T10:44:24.970 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T10:44:24.970 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T10:44:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:44:24.987 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-16T10:44:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:44:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:44:24.988 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-16T10:44:24.989 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-16T10:44:25.044 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-16T10:44:25.044 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-03-16T10:44:25.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-16T10:44:25.045 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-03-16T10:44:25.046 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-03-16T10:44:25.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-03-16T10:44:25.047 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-361.el9.x86_64 2024-03-16T10:44:25.047 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-361.el9.x86_64 2024-03-16T10:44:25.047 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-03-16T10:44:27.109 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-16T10:44:27.229 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-16T10:44:27.235 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-16T10:44:28.685 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-16T10:44:28.829 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-16T10:44:28.852 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-16T10:44:29.636 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-16T10:44:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-16T10:44:29.885 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-16T10:44:31.580 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-16T10:44:31.685 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-16T10:44:33.644 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-03-16T10:44:35.564 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-16T10:44:35.636 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-16T10:44:37.070 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2024-03-16T10:44:37.089 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-03-16T10:44:37.736 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-16T10:44:37.974 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-03-16T10:44:38.507 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-16T10:44:39.062 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 15 kB 00:00 2024-03-16T10:44:40.482 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-16T10:44:43.008 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-16T10:44:43.521 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-16T10:44:45.181 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T10:44:45.181 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T10:44:45.273 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-16T10:44:45.280 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-16T10:44:45.281 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-03-16T10:44:45.281 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:44:45.281 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 100 M 2024-03-16T10:44:45.281 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-03-16T10:44:45.281 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-16T10:44:45.405 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-16T10:44:45.737 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T10:44:45.737 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T10:44:45.801 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 16 MB/s | 6.0 MB 00:00 2024-03-16T10:44:45.833 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-16T10:44:45.849 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-16T10:44:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-03-16T10:44:45.849 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-16T10:44:45.849 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-16T10:44:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T10:44:45.849 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-03-16T10:44:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:44:45.851 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 100 M 2024-03-16T10:44:45.851 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 123 M 2024-03-16T10:44:45.851 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-16T10:44:46.335 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-03-16T10:44:46.590 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-03-16T10:44:46.985 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 25 MB/s | 39 MB 00:01 2024-03-16T10:44:47.098 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 20 MB/s | 21 MB 00:01 2024-03-16T10:44:47.603 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T10:44:47.603 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T10:44:47.648 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 32 MB/s | 33 MB 00:01 2024-03-16T10:44:47.677 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 18 MB/s | 33 MB 00:01 2024-03-16T10:44:47.677 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:44:47.677 INFO:teuthology.orchestra.run.smithi098.stdout:Total 42 MB/s | 100 MB 00:02 2024-03-16T10:44:47.690 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-16T10:44:47.701 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-16T10:44:47.701 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-16T10:44:47.701 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-16T10:44:47.701 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-16T10:44:47.701 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T10:44:47.701 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-03-16T10:44:47.702 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:44:47.703 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 100 M 2024-03-16T10:44:47.703 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-03-16T10:44:47.703 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-16T10:44:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-03-16T10:44:48.410 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:44:48.410 INFO:teuthology.orchestra.run.smithi078.stdout:Total 39 MB/s | 100 MB 00:02 2024-03-16T10:44:48.560 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-16T10:44:48.935 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-16T10:44:48.935 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-16T10:44:49.295 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-16T10:44:49.661 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-16T10:44:49.661 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-16T10:44:50.314 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-16T10:44:50.314 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-16T10:44:51.325 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-16T10:44:51.326 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-16T10:44:51.389 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 1.7 MB/s | 6.0 MB 00:03 2024-03-16T10:44:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-16T10:44:53.942 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T10:44:54.028 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-16T10:44:54.063 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:44:54.680 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T10:44:54.808 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:44:56.844 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:44:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:44:57.600 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:44:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:45:01.744 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:45:01.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:02.549 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:45:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:06.770 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.1 MB/s | 21 MB 00:19 2024-03-16T10:45:07.400 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:24.693 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 1.1 MB/s | 39 MB 00:36 2024-03-16T10:45:31.376 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 856 kB/s | 33 MB 00:39 2024-03-16T10:45:31.377 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:45:31.377 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.3 MB/s | 100 MB 00:43 2024-03-16T10:45:32.259 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-16T10:45:32.641 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-16T10:45:32.641 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-16T10:45:34.263 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-16T10:45:34.263 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-16T10:45:36.393 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:36.393 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T10:45:36.394 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:45:36.477 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:36.493 INFO:teuthology.orchestra.run.smithi078.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T10:45:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:45:36.578 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T10:45:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:45:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:45:36.968 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-16T10:45:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T10:45:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:45:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:45:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:37.276 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:45:37.277 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-16T10:45:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.277 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:45:37.277 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-16T10:45:37.482 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:37.482 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:45:37.482 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-16T10:45:37.482 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.482 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.482 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.483 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.483 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:45:37.483 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-16T10:45:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T10:45:37.742 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:45:37.814 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T10:45:37.956 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T10:45:37.956 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T10:45:37.956 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T10:45:37.957 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T10:45:37.957 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T10:45:37.957 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-16T10:45:37.957 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T10:45:37.957 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T10:45:38.019 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T10:45:38.077 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T10:45:38.078 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T10:45:38.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-16T10:45:38.120 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-16T10:45:38.120 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-16T10:45:38.120 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-16T10:45:38.120 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T10:45:38.121 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:45:38.122 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T10:45:38.123 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T10:45:38.123 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-427.el9.x86_64 2024-03-16T10:45:38.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T10:45:38.208 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T10:45:38.208 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T10:45:38.208 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T10:45:38.209 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T10:45:38.209 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T10:45:38.209 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-16T10:45:38.209 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T10:45:38.209 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T10:45:38.330 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T10:45:38.330 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T10:45:38.331 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-03-16T10:45:38.373 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-03-16T10:45:38.373 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-03-16T10:45:38.373 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-03-16T10:45:38.373 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 68.el9 2024-03-16T10:45:38.373 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-03-16T10:45:38.373 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8264254 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:45:38.374 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T10:45:38.375 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T10:45:38.375 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T10:45:38.375 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-03-16T10:45:38.375 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:45:38.375 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T10:45:38.376 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T10:45:38.376 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-427.el9.x86_64 2024-03-16T10:45:38.376 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T10:45:38.711 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-16T10:45:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-03-16T10:45:40.158 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T10:45:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-16T10:45:40.194 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-16T10:45:40.209 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.MEn9mqBfT3 2024-03-16T10:45:40.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MEn9mqBfT3 2024-03-16T10:45:40.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.MEn9mqBfT3 2024-03-16T10:45:40.424 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.MEn9mqBfT3 is 6KB 2024-03-16T10:45:40.494 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.MEn9mqBfT3 2024-03-16T10:45:40.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T10:45:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T10:45:40.548 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-03-16T10:45:40.550 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-03-16T10:45:40.552 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:45:40.593 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T10:45:40.593 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T10:45:40.594 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T10:45:40.594 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T10:45:40.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T10:45:40.620 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.FPaQGeSm6X 2024-03-16T10:45:40.620 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FPaQGeSm6X 2024-03-16T10:45:40.697 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.FPaQGeSm6X 2024-03-16T10:45:40.737 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:45:40.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-16T10:45:40.847 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.FPaQGeSm6X is 6KB 2024-03-16T10:45:40.870 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.FPaQGeSm6X 2024-03-16T10:45:40.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T10:45:40.954 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T10:45:40.955 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T10:45:40.955 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T10:45:40.955 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T10:45:40.956 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T10:45:41.130 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-03-16T10:45:45.453 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:45:45.580 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:45:50.343 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:46:10.757 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-16T10:46:10.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T10:46:10.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:46:11.133 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-16T10:46:11.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-16T10:46:11.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:46:19.680 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:46:19.681 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T10:46:19.681 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:46:19.763 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:46:20.149 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:46:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T10:46:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T10:46:20.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T10:46:20.593 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T10:46:20.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:46:20.593 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-16T10:46:20.593 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T10:46:20.593 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T10:46:20.593 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T10:46:20.594 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T10:46:20.594 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:46:20.594 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-16T10:46:21.107 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T10:46:21.247 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T10:46:21.248 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T10:46:21.248 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T10:46:21.249 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T10:46:21.249 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T10:46:21.249 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-16T10:46:21.249 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T10:46:21.249 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T10:46:21.370 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T10:46:21.370 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T10:46:21.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-03-16T10:46:21.404 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-03-16T10:46:21.404 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-03-16T10:46:21.404 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T10:46:21.405 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:46:21.406 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T10:46:21.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T10:46:21.407 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-427.el9.x86_64 2024-03-16T10:46:21.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T10:46:22.087 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-16T10:46:23.537 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T10:46:23.558 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-16T10:46:23.561 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-16T10:46:23.576 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.h3Lrapg2FU 2024-03-16T10:46:23.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h3Lrapg2FU 2024-03-16T10:46:23.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.h3Lrapg2FU 2024-03-16T10:46:23.787 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.h3Lrapg2FU is 6KB 2024-03-16T10:46:23.834 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.h3Lrapg2FU 2024-03-16T10:46:23.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T10:46:23.921 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T10:46:23.922 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T10:46:23.922 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T10:46:23.922 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T10:46:23.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T10:46:24.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-16T10:46:29.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-16T10:46:29.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-03-16T10:46:38.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T10:46:38.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:46:38.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-16T10:46:38.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:46:41.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-16T10:46:41.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-03-16T10:46:53.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T10:46:53.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:46:53.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-16T10:46:53.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:46:54.084 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-16T10:46:54.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T10:46:54.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:47:11.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-16T10:47:12.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-03-16T10:47:12.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-16T10:47:21.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T10:47:21.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:47:24.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-16T10:47:26.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T10:47:26.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:47:27.059 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-16T10:47:27.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-16T10:47:27.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:47:27.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T10:47:27.470 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-16T10:47:27.471 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-16T10:47:27.484 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-427.el9.x86_64 2024-03-16T10:47:27.484 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T10:47:27.484 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T10:47:27.485 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T10:47:27.950 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-16T10:47:28.437 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-16T10:47:28.437 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-16T10:47:28.438 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-16T10:47:28.463 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-427.el9.x86_64 2024-03-16T10:47:28.463 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T10:47:28.463 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T10:47:28.463 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T10:47:28.485 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-16T10:47:28.485 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T10:47:28.575 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-03-16T10:47:28.621 DEBUG:teuthology.parallel:result is None 2024-03-16T10:47:29.464 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-16T10:47:29.464 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T10:47:29.520 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-03-16T10:47:29.627 DEBUG:teuthology.parallel:result is None 2024-03-16T10:47:36.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T10:47:36.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:48:36.483 DEBUG:teuthology.orchestra.remote:timed out 2024-03-16T10:48:51.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T10:48:51.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:48:51.830 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-16T10:48:52.220 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T10:48:52.220 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-16T10:48:52.220 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-16T10:48:52.236 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-427.el9.x86_64 2024-03-16T10:48:52.236 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T10:48:52.236 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T10:48:52.236 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T10:48:53.236 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-16T10:48:53.237 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T10:48:53.321 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-16T10:48:53.354 DEBUG:teuthology.parallel:result is None 2024-03-16T10:48:53.354 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T10:48:53.361 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T10:48:53.361 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T10:48:53.364 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T10:48:53.367 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T10:48:53.387 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T10:48:53.393 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T10:48:53.400 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T10:48:53.400 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T10:48:53.422 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T10:48:53.425 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T10:48:53.461 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T10:48:53.468 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T10:48:53.468 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T10:48:53.482 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T10:48:53.489 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T10:48:53.521 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T10:48:53.523 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T10:48:53.532 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T10:48:53.535 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T10:48:53.537 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T10:48:53.546 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T10:48:53.548 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T10:48:53.555 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T10:48:53.555 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T10:48:53.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T10:48:53.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T10:48:53.619 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T10:48:53.628 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T10:48:53.628 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T10:48:53.648 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T10:48:53.652 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T10:48:53.675 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T10:48:53.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T10:48:53.806 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T10:48:53.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T10:48:53.888 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:48:53.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T10:48:53.955 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T10:48:55.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T10:48:55.679 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T10:48:55.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T10:48:55.767 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:48:55.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T10:48:55.835 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T10:48:59.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T10:48:59.461 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T10:48:59.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T10:48:59.592 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:48:59.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T10:48:59.662 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-03-16T10:48:59.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-16T10:48:59.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-16T10:48:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T10:48:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T10:48:59.745 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T10:49:00.101 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T10:49:00.109 INFO:teuthology.task.internal:Starting timer... 2024-03-16T10:49:00.109 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T10:49:00.119 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T10:49:00.127 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-16T10:49:00.128 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-03-16T10:49:00.207 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-03-16T10:49:00.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-16T10:49:00.275 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-16T10:49:00.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-16T10:49:00.341 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-16T10:49:00.343 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T10:49:00.343 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-03-16T10:49:00.375 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-03-16T10:49:00.375 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-16T10:49:00.405 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-16T10:49:00.405 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-16T10:49:00.431 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-16T10:49:00.431 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-03-16T10:49:00.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T10:49:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:49:00.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T10:49:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:49:00.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T10:49:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:49:00.520 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T10:49:00.520 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-03-16T10:49:00.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-16T10:49:00.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-16T10:49:00.633 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T10:49:00.643 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-16T10:49:00.648 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-16T10:49:00.758 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T10:49:00.772 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T10:49:00.774 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi098.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T10:53:11.096 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-03-16T10:53:11.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-16T10:53:11.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:53:11.207 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-16T10:53:11.289 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-16T10:53:11.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T10:53:11.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:53:11.377 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-16T10:53:11.464 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T10:53:11.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T10:53:11.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:53:11.551 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-16T10:53:11.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T10:53:11.625 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T10:53:11.636 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T10:53:11.636 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T10:53:11.637 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:53:11.639 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T10:53:11.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:53:11.641 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T10:53:11.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:53:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T10:53:11.671 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T10:53:11.686 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T10:53:11.688 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T10:53:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T10:53:11.712 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T10:53:11.716 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-03-16T10:53:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-16T10:53:11.729 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-03-16T10:53:11.730 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-16T10:53:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-16T10:53:11.746 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T10:53:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T10:53:11.755 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-16T10:53:11.762 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T10:53:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T10:53:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T10:53:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T10:53:11.813 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:53:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:53:11.818 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:53:11.818 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-03-16T10:53:11.818 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.818 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.818 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.818 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.819 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:53:11.819 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-16T10:53:11.819 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.819 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.820 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.820 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:53:11.841 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:53:11.841 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-16T10:53:11.841 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.841 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.841 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.842 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:53:11.842 INFO:teuthology.run_tasks:Running task install... 2024-03-16T10:53:11.853 DEBUG:teuthology.task.install:project ceph 2024-03-16T10:53:11.853 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-16T10:53:11.853 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-16T10:53:11.853 INFO:teuthology.task.install:Using flavor: default 2024-03-16T10:53:11.859 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-16T10:53:11.859 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T10:53:11.859 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T10:53:11.860 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:53:11.862 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T10:53:11.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:53:11.863 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T10:53:11.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:53:12.084 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T10:53:12.085 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T10:53:12.088 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T10:53:12.088 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T10:53:12.092 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T10:53:12.092 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T10:53:12.198 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T10:53:12.198 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:53:12.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T10:53:12.204 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T10:53:12.205 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:53:12.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T10:53:12.225 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T10:53:12.226 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T10:53:12.233 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T10:53:12.233 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T10:53:12.254 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T10:53:12.254 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:53:12.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T10:53:12.283 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T10:53:12.283 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T10:53:12.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T10:53:12.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T10:53:12.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T10:53:12.372 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T10:53:12.381 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T10:53:12.431 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T10:53:12.439 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-16T10:53:12.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-16T10:53:12.448 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-16T10:53:12.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-16T10:53:12.498 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-03-16T10:53:12.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-03-16T10:53:12.683 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-16T10:53:12.705 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-16T10:53:12.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T10:53:12.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T10:53:12.743 INFO:teuthology.orchestra.run.smithi078.stdout:74 files removed 2024-03-16T10:53:12.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T10:53:13.402 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 223 kB/s | 83 kB 00:00 2024-03-16T10:53:13.454 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 206 kB/s | 83 kB 00:00 2024-03-16T10:53:13.488 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 207 kB/s | 83 kB 00:00 2024-03-16T10:53:13.719 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 37 kB/s | 10 kB 00:00 2024-03-16T10:53:13.805 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 37 kB/s | 10 kB 00:00 2024-03-16T10:53:13.846 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-16T10:53:14.070 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-16T10:53:14.181 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-16T10:53:14.197 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-16T10:53:14.464 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-16T10:53:14.642 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-16T10:53:14.899 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-16T10:53:15.865 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-16T10:53:16.134 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-16T10:53:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-16T10:53:16.747 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-16T10:53:17.183 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-16T10:53:17.274 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-16T10:53:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-16T10:53:19.692 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-16T10:53:21.223 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-16T10:53:23.268 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.1 MB 00:00 2024-03-16T10:53:24.004 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-16T10:53:24.861 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-16T10:53:25.182 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-16T10:53:25.861 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-16T10:53:26.096 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-16T10:53:26.758 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-16T10:53:27.546 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-16T10:53:27.968 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-16T10:53:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-16T10:53:32.446 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-16T10:53:32.868 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-16T10:53:33.618 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:53:33.619 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:53:33.633 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T10:53:33.634 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T10:53:33.635 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T10:53:33.636 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T10:53:33.638 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T10:53:33.641 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T10:53:33.642 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T10:53:33.722 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T10:53:33.750 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T10:53:33.751 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T10:53:33.752 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T10:53:33.753 INFO:teuthology.orchestra.run.smithi078.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T10:53:33.754 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T10:53:33.755 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T10:53:33.756 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T10:53:33.757 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T10:53:33.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T10:53:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T10:53:33.760 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T10:53:33.761 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T10:53:33.762 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:Install 150 Packages 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 24 Packages 2024-03-16T10:53:33.763 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:53:33.764 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 224 M 2024-03-16T10:53:33.764 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-16T10:53:35.001 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:53:35.002 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:53:35.015 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T10:53:35.017 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T10:53:35.018 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T10:53:35.018 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T10:53:35.021 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T10:53:35.023 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T10:53:35.024 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T10:53:35.088 INFO:teuthology.orchestra.run.smithi078.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-16T10:53:35.104 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-16T10:53:35.122 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-16T10:53:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-16T10:53:35.122 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T10:53:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T10:53:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T10:53:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T10:53:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T10:53:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T10:53:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T10:53:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T10:53:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T10:53:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T10:53:35.132 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T10:53:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T10:53:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T10:53:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T10:53:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout:Install 150 Packages 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 24 Packages 2024-03-16T10:53:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:53:35.138 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 224 M 2024-03-16T10:53:35.138 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-16T10:53:35.330 INFO:teuthology.orchestra.run.smithi078.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 3.6 MB/s | 890 kB 00:00 2024-03-16T10:53:35.398 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:53:35.399 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:53:35.413 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T10:53:35.414 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T10:53:35.415 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T10:53:35.416 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T10:53:35.418 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T10:53:35.421 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T10:53:35.422 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T10:53:35.447 INFO:teuthology.orchestra.run.smithi078.stdout:(3/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 6.7 MB/s | 5.4 MB 00:00 2024-03-16T10:53:35.472 INFO:teuthology.orchestra.run.smithi078.stdout:(4/174): ceph-immutable-object-cache-19.0.0-216 1.0 MB/s | 145 kB 00:00 2024-03-16T10:53:35.502 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T10:53:35.518 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T10:53:35.519 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T10:53:35.520 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T10:53:35.521 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T10:53:35.522 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T10:53:35.523 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T10:53:35.524 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T10:53:35.525 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T10:53:35.526 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T10:53:35.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T10:53:35.528 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T10:53:35.529 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout:Install 150 Packages 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 24 Packages 2024-03-16T10:53:35.530 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:53:35.531 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 224 M 2024-03-16T10:53:35.531 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-16T10:53:35.606 INFO:teuthology.orchestra.run.smithi078.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 15 MB/s | 2.4 MB 00:00 2024-03-16T10:53:35.715 INFO:teuthology.orchestra.run.smithi078.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-03-16T10:53:35.898 INFO:teuthology.orchestra.run.smithi078.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 16 MB/s | 4.7 MB 00:00 2024-03-16T10:53:36.494 INFO:teuthology.orchestra.run.smithi153.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-16T10:53:36.514 INFO:teuthology.orchestra.run.smithi098.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-16T10:53:36.532 INFO:teuthology.orchestra.run.smithi078.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 16 MB/s | 9.8 MB 00:00 2024-03-16T10:53:36.590 INFO:teuthology.orchestra.run.smithi078.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 432 kB/s | 25 kB 00:00 2024-03-16T10:53:37.510 INFO:teuthology.orchestra.run.smithi153.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 876 kB/s | 890 kB 00:01 2024-03-16T10:53:37.661 INFO:teuthology.orchestra.run.smithi153.stdout:(3/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 3.6 MB/s | 5.4 MB 00:01 2024-03-16T10:53:37.723 INFO:teuthology.orchestra.run.smithi098.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 737 kB/s | 890 kB 00:01 2024-03-16T10:53:37.736 INFO:teuthology.orchestra.run.smithi153.stdout:(4/174): ceph-immutable-object-cache-19.0.0-216 643 kB/s | 145 kB 00:00 2024-03-16T10:53:37.956 INFO:teuthology.orchestra.run.smithi098.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 621 kB/s | 145 kB 00:00 2024-03-16T10:53:38.149 INFO:teuthology.orchestra.run.smithi078.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 6.9 MB/s | 17 MB 00:02 2024-03-16T10:53:38.211 INFO:teuthology.orchestra.run.smithi153.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 4.3 MB/s | 2.4 MB 00:00 2024-03-16T10:53:38.241 INFO:teuthology.orchestra.run.smithi078.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 221 kB 00:00 2024-03-16T10:53:38.332 INFO:teuthology.orchestra.run.smithi078.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 296 kB/s | 27 kB 00:00 2024-03-16T10:53:38.424 INFO:teuthology.orchestra.run.smithi078.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 345 kB/s | 32 kB 00:00 2024-03-16T10:53:38.608 INFO:teuthology.orchestra.run.smithi078.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.0 MB/s | 747 kB 00:00 2024-03-16T10:53:38.692 INFO:teuthology.orchestra.run.smithi078.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.9 MB/s | 163 kB 00:00 2024-03-16T10:53:38.783 INFO:teuthology.orchestra.run.smithi078.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 127 kB 00:00 2024-03-16T10:53:38.942 INFO:teuthology.orchestra.run.smithi078.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 3.1 MB/s | 501 kB 00:00 2024-03-16T10:53:39.161 INFO:teuthology.orchestra.run.smithi153.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 1.0 MB/s | 1.5 MB 00:01 2024-03-16T10:53:39.245 INFO:teuthology.orchestra.run.smithi153.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 4.5 MB/s | 4.7 MB 00:01 2024-03-16T10:53:39.726 INFO:teuthology.orchestra.run.smithi078.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 6.6 MB/s | 5.1 MB 00:00 2024-03-16T10:53:39.731 INFO:teuthology.orchestra.run.smithi098.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-03-16T10:53:39.792 INFO:teuthology.orchestra.run.smithi078.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 680 kB/s | 45 kB 00:00 2024-03-16T10:53:39.868 INFO:teuthology.orchestra.run.smithi078.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.7 MB/s | 131 kB 00:00 2024-03-16T10:53:39.951 INFO:teuthology.orchestra.run.smithi078.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-16T10:53:40.043 INFO:teuthology.orchestra.run.smithi078.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.4 MB/s | 320 kB 00:00 2024-03-16T10:53:40.127 INFO:teuthology.orchestra.run.smithi078.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.5 MB/s | 298 kB 00:00 2024-03-16T10:53:40.202 INFO:teuthology.orchestra.run.smithi078.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 99 kB 00:00 2024-03-16T10:53:40.215 INFO:teuthology.orchestra.run.smithi098.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 3.1 MB/s | 1.5 MB 00:00 2024-03-16T10:53:40.261 INFO:teuthology.orchestra.run.smithi078.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-16T10:53:40.273 INFO:teuthology.orchestra.run.smithi098.stdout:(6/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.0 MB/s | 2.4 MB 00:02 2024-03-16T10:53:40.778 INFO:teuthology.orchestra.run.smithi078.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 6.0 MB/s | 3.1 MB 00:00 2024-03-16T10:53:40.870 INFO:teuthology.orchestra.run.smithi078.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-16T10:53:40.962 INFO:teuthology.orchestra.run.smithi078.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 371 kB/s | 34 kB 00:00 2024-03-16T10:53:41.054 INFO:teuthology.orchestra.run.smithi078.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-16T10:53:41.157 INFO:teuthology.orchestra.run.smithi098.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 5.0 MB/s | 4.7 MB 00:00 2024-03-16T10:53:41.371 INFO:teuthology.orchestra.run.smithi078.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 5.7 MB/s | 1.8 MB 00:00 2024-03-16T10:53:41.870 INFO:teuthology.orchestra.run.smithi153.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 3.7 MB/s | 9.8 MB 00:02 2024-03-16T10:53:41.962 INFO:teuthology.orchestra.run.smithi153.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 275 kB/s | 25 kB 00:00 2024-03-16T10:53:42.371 INFO:teuthology.orchestra.run.smithi078.stdout:(31/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 8.2 MB/s | 48 MB 00:05 2024-03-16T10:53:42.479 INFO:teuthology.orchestra.run.smithi078.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 6.7 MB/s | 7.4 MB 00:01 2024-03-16T10:53:42.505 INFO:teuthology.orchestra.run.smithi078.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.8 MB/s | 248 kB 00:00 2024-03-16T10:53:42.546 INFO:teuthology.orchestra.run.smithi078.stdout:(34/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 740 kB/s | 49 kB 00:00 2024-03-16T10:53:42.572 INFO:teuthology.orchestra.run.smithi078.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 246 kB/s | 16 kB 00:00 2024-03-16T10:53:42.622 INFO:teuthology.orchestra.run.smithi078.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.4 MB/s | 264 kB 00:00 2024-03-16T10:53:42.714 INFO:teuthology.orchestra.run.smithi078.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 199 kB/s | 18 kB 00:00 2024-03-16T10:53:42.747 INFO:teuthology.orchestra.run.smithi078.stdout:(38/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-16T10:53:42.882 INFO:teuthology.orchestra.run.smithi098.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 5.7 MB/s | 9.8 MB 00:01 2024-03-16T10:53:42.898 INFO:teuthology.orchestra.run.smithi078.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-16T10:53:42.932 INFO:teuthology.orchestra.run.smithi098.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 506 kB/s | 25 kB 00:00 2024-03-16T10:53:42.940 INFO:teuthology.orchestra.run.smithi078.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.5 MB/s | 920 kB 00:00 2024-03-16T10:53:42.965 INFO:teuthology.orchestra.run.smithi078.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-16T10:53:42.990 INFO:teuthology.orchestra.run.smithi078.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-16T10:53:43.015 INFO:teuthology.orchestra.run.smithi078.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-03-16T10:53:43.079 INFO:teuthology.orchestra.run.smithi153.stdout:(10/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 3.0 MB/s | 21 MB 00:06 2024-03-16T10:53:43.107 INFO:teuthology.orchestra.run.smithi078.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-16T10:53:43.133 INFO:teuthology.orchestra.run.smithi078.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 149 kB/s | 17 kB 00:00 2024-03-16T10:53:43.158 INFO:teuthology.orchestra.run.smithi078.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-16T10:53:43.179 INFO:teuthology.orchestra.run.smithi153.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.2 MB/s | 221 kB 00:00 2024-03-16T10:53:43.183 INFO:teuthology.orchestra.run.smithi078.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-16T10:53:43.209 INFO:teuthology.orchestra.run.smithi078.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-16T10:53:43.229 INFO:teuthology.orchestra.run.smithi153.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 544 kB/s | 27 kB 00:00 2024-03-16T10:53:43.234 INFO:teuthology.orchestra.run.smithi078.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T10:53:43.268 INFO:teuthology.orchestra.run.smithi078.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-16T10:53:43.279 INFO:teuthology.orchestra.run.smithi153.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 635 kB/s | 32 kB 00:00 2024-03-16T10:53:43.293 INFO:teuthology.orchestra.run.smithi078.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 46 kB 00:00 2024-03-16T10:53:43.318 INFO:teuthology.orchestra.run.smithi078.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-16T10:53:43.343 INFO:teuthology.orchestra.run.smithi078.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-16T10:53:43.369 INFO:teuthology.orchestra.run.smithi078.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-16T10:53:43.394 INFO:teuthology.orchestra.run.smithi078.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-16T10:53:43.471 INFO:teuthology.orchestra.run.smithi153.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 3.8 MB/s | 747 kB 00:00 2024-03-16T10:53:43.494 INFO:teuthology.orchestra.run.smithi078.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-16T10:53:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-16T10:53:43.555 INFO:teuthology.orchestra.run.smithi153.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.9 MB/s | 163 kB 00:00 2024-03-16T10:53:43.570 INFO:teuthology.orchestra.run.smithi078.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-16T10:53:43.595 INFO:teuthology.orchestra.run.smithi078.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-16T10:53:43.622 INFO:teuthology.orchestra.run.smithi153.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-16T10:53:43.720 INFO:teuthology.orchestra.run.smithi078.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-16T10:53:43.754 INFO:teuthology.orchestra.run.smithi078.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-16T10:53:43.764 INFO:teuthology.orchestra.run.smithi153.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 3.5 MB/s | 501 kB 00:00 2024-03-16T10:53:43.779 INFO:teuthology.orchestra.run.smithi078.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-16T10:53:43.896 INFO:teuthology.orchestra.run.smithi078.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-16T10:53:43.921 INFO:teuthology.orchestra.run.smithi078.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 176 kB/s | 24 kB 00:00 2024-03-16T10:53:43.947 INFO:teuthology.orchestra.run.smithi078.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-16T10:53:43.972 INFO:teuthology.orchestra.run.smithi078.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-16T10:53:43.997 INFO:teuthology.orchestra.run.smithi078.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-16T10:53:44.106 INFO:teuthology.orchestra.run.smithi078.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-16T10:53:44.131 INFO:teuthology.orchestra.run.smithi078.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-16T10:53:44.157 INFO:teuthology.orchestra.run.smithi078.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-16T10:53:44.182 INFO:teuthology.orchestra.run.smithi078.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-16T10:53:44.207 INFO:teuthology.orchestra.run.smithi078.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2024-03-16T10:53:44.232 INFO:teuthology.orchestra.run.smithi078.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-16T10:53:44.258 INFO:teuthology.orchestra.run.smithi078.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-16T10:53:44.283 INFO:teuthology.orchestra.run.smithi078.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-16T10:53:44.300 INFO:teuthology.orchestra.run.smithi078.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-16T10:53:44.317 INFO:teuthology.orchestra.run.smithi078.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-16T10:53:44.467 INFO:teuthology.orchestra.run.smithi078.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 259 kB/s | 39 kB 00:00 2024-03-16T10:53:44.709 INFO:teuthology.orchestra.run.smithi078.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-16T10:53:44.734 INFO:teuthology.orchestra.run.smithi078.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 154 kB/s | 41 kB 00:00 2024-03-16T10:53:44.759 INFO:teuthology.orchestra.run.smithi078.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-16T10:53:44.785 INFO:teuthology.orchestra.run.smithi078.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-16T10:53:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-16T10:53:44.835 INFO:teuthology.orchestra.run.smithi078.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-16T10:53:44.860 INFO:teuthology.orchestra.run.smithi078.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-16T10:53:44.886 INFO:teuthology.orchestra.run.smithi078.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-16T10:53:44.928 INFO:teuthology.orchestra.run.smithi078.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-16T10:53:44.953 INFO:teuthology.orchestra.run.smithi078.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-16T10:53:44.956 INFO:teuthology.orchestra.run.smithi153.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 4.3 MB/s | 5.1 MB 00:01 2024-03-16T10:53:44.978 INFO:teuthology.orchestra.run.smithi078.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-16T10:53:45.004 INFO:teuthology.orchestra.run.smithi078.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-16T10:53:45.022 INFO:teuthology.orchestra.run.smithi153.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 679 kB/s | 45 kB 00:00 2024-03-16T10:53:45.071 INFO:teuthology.orchestra.run.smithi078.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-16T10:53:45.098 INFO:teuthology.orchestra.run.smithi153.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.7 MB/s | 131 kB 00:00 2024-03-16T10:53:45.129 INFO:teuthology.orchestra.run.smithi078.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-03-16T10:53:45.165 INFO:teuthology.orchestra.run.smithi153.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.4 MB/s | 162 kB 00:00 2024-03-16T10:53:45.171 INFO:teuthology.orchestra.run.smithi078.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-03-16T10:53:45.230 INFO:teuthology.orchestra.run.smithi078.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-03-16T10:53:45.282 INFO:teuthology.orchestra.run.smithi153.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.7 MB/s | 320 kB 00:00 2024-03-16T10:53:45.364 INFO:teuthology.orchestra.run.smithi078.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 112 kB/s | 21 kB 00:00 2024-03-16T10:53:45.389 INFO:teuthology.orchestra.run.smithi078.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 160 kB/s | 25 kB 00:00 2024-03-16T10:53:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.5 MB/s | 298 kB 00:00 2024-03-16T10:53:45.465 INFO:teuthology.orchestra.run.smithi153.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 99 kB 00:00 2024-03-16T10:53:45.506 INFO:teuthology.orchestra.run.smithi078.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-16T10:53:45.533 INFO:teuthology.orchestra.run.smithi153.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-16T10:53:45.614 INFO:teuthology.orchestra.run.smithi078.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2024-03-16T10:53:45.640 INFO:teuthology.orchestra.run.smithi078.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-03-16T10:53:45.681 INFO:teuthology.orchestra.run.smithi078.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-16T10:53:45.707 INFO:teuthology.orchestra.run.smithi078.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 522 kB/s | 35 kB 00:00 2024-03-16T10:53:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 664 kB/s | 44 kB 00:00 2024-03-16T10:53:45.774 INFO:teuthology.orchestra.run.smithi078.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-16T10:53:46.316 INFO:teuthology.orchestra.run.smithi078.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-16T10:53:46.333 INFO:teuthology.orchestra.run.smithi153.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 3.9 MB/s | 3.1 MB 00:00 2024-03-16T10:53:46.408 INFO:teuthology.orchestra.run.smithi153.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-16T10:53:46.483 INFO:teuthology.orchestra.run.smithi153.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 452 kB/s | 34 kB 00:00 2024-03-16T10:53:46.499 INFO:teuthology.orchestra.run.smithi078.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.9 MB/s | 548 kB 00:00 2024-03-16T10:53:46.550 INFO:teuthology.orchestra.run.smithi078.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 6.0 MB/s | 309 kB 00:00 2024-03-16T10:53:46.550 INFO:teuthology.orchestra.run.smithi153.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.0 MB/s | 139 kB 00:00 2024-03-16T10:53:46.691 INFO:teuthology.orchestra.run.smithi098.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 2.6 MB/s | 17 MB 00:06 2024-03-16T10:53:46.767 INFO:teuthology.orchestra.run.smithi078.stdout:(107/174): ceph-common-19.0.0-2162.g3d78f5ba.el 1.7 MB/s | 21 MB 00:12 2024-03-16T10:53:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-16T10:53:46.834 INFO:teuthology.orchestra.run.smithi078.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-03-16T10:53:46.859 INFO:teuthology.orchestra.run.smithi153.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 5.8 MB/s | 1.8 MB 00:00 2024-03-16T10:53:46.876 INFO:teuthology.orchestra.run.smithi078.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-16T10:53:46.882 INFO:teuthology.orchestra.run.smithi098.stdout:(11/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 1.9 MB/s | 21 MB 00:10 2024-03-16T10:53:46.908 INFO:teuthology.orchestra.run.smithi098.stdout:(12/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.0 MB/s | 221 kB 00:00 2024-03-16T10:53:46.917 INFO:teuthology.orchestra.run.smithi078.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-16T10:53:46.958 INFO:teuthology.orchestra.run.smithi098.stdout:(13/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 360 kB/s | 27 kB 00:00 2024-03-16T10:53:46.968 INFO:teuthology.orchestra.run.smithi078.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-16T10:53:46.975 INFO:teuthology.orchestra.run.smithi098.stdout:(14/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 475 kB/s | 32 kB 00:00 2024-03-16T10:53:47.035 INFO:teuthology.orchestra.run.smithi078.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-16T10:53:47.042 INFO:teuthology.orchestra.run.smithi098.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 163 kB 00:00 2024-03-16T10:53:47.094 INFO:teuthology.orchestra.run.smithi078.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-16T10:53:47.109 INFO:teuthology.orchestra.run.smithi098.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-16T10:53:47.119 INFO:teuthology.orchestra.run.smithi078.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-16T10:53:47.144 INFO:teuthology.orchestra.run.smithi078.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 867 kB/s | 43 kB 00:00 2024-03-16T10:53:47.161 INFO:teuthology.orchestra.run.smithi078.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 766 kB/s | 32 kB 00:00 2024-03-16T10:53:47.178 INFO:teuthology.orchestra.run.smithi078.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 445 kB/s | 15 kB 00:00 2024-03-16T10:53:47.204 INFO:teuthology.orchestra.run.smithi078.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-03-16T10:53:47.229 INFO:teuthology.orchestra.run.smithi078.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-16T10:53:47.254 INFO:teuthology.orchestra.run.smithi078.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-16T10:53:47.276 INFO:teuthology.orchestra.run.smithi098.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.9 MB/s | 501 kB 00:00 2024-03-16T10:53:47.279 INFO:teuthology.orchestra.run.smithi078.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 705 kB/s | 35 kB 00:00 2024-03-16T10:53:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-16T10:53:47.363 INFO:teuthology.orchestra.run.smithi078.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 24 MB/s | 2.0 MB 00:00 2024-03-16T10:53:47.367 INFO:teuthology.orchestra.run.smithi098.stdout:(18/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 1.8 MB/s | 747 kB 00:00 2024-03-16T10:53:47.388 INFO:teuthology.orchestra.run.smithi078.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-16T10:53:47.414 INFO:teuthology.orchestra.run.smithi078.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-03-16T10:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-03-16T10:53:47.464 INFO:teuthology.orchestra.run.smithi078.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-16T10:53:47.485 INFO:teuthology.orchestra.run.smithi098.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 386 kB/s | 45 kB 00:00 2024-03-16T10:53:47.489 INFO:teuthology.orchestra.run.smithi078.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-03-16T10:53:47.515 INFO:teuthology.orchestra.run.smithi078.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-16T10:53:47.548 INFO:teuthology.orchestra.run.smithi078.stdout:(131/174): protobuf-compiler-3.14.0-13.el9.x86_ 487 kB/s | 863 kB 00:01 2024-03-16T10:53:47.568 INFO:teuthology.orchestra.run.smithi098.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-16T10:53:47.596 INFO:teuthology.orchestra.run.smithi078.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 571 kB/s | 46 kB 00:00 2024-03-16T10:53:47.632 INFO:teuthology.orchestra.run.smithi078.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-16T10:53:47.652 INFO:teuthology.orchestra.run.smithi098.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-16T10:53:47.657 INFO:teuthology.orchestra.run.smithi078.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 950 kB/s | 58 kB 00:00 2024-03-16T10:53:47.682 INFO:teuthology.orchestra.run.smithi078.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-16T10:53:47.692 INFO:teuthology.orchestra.run.smithi153.stdout:(31/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 2.0 MB/s | 17 MB 00:08 2024-03-16T10:53:47.708 INFO:teuthology.orchestra.run.smithi078.stdout:(136/174): python3-more-itertools-8.12.0-2.el9. 496 kB/s | 79 kB 00:00 2024-03-16T10:53:47.733 INFO:teuthology.orchestra.run.smithi078.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-16T10:53:47.758 INFO:teuthology.orchestra.run.smithi078.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-16T10:53:47.784 INFO:teuthology.orchestra.run.smithi078.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-03-16T10:53:47.802 INFO:teuthology.orchestra.run.smithi098.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 320 kB 00:00 2024-03-16T10:53:47.808 INFO:teuthology.orchestra.run.smithi078.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-03-16T10:53:47.834 INFO:teuthology.orchestra.run.smithi078.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-16T10:53:47.843 INFO:teuthology.orchestra.run.smithi153.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.6 MB/s | 248 kB 00:00 2024-03-16T10:53:47.859 INFO:teuthology.orchestra.run.smithi078.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-16T10:53:47.884 INFO:teuthology.orchestra.run.smithi078.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-03-16T10:53:47.901 INFO:teuthology.orchestra.run.smithi153.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 849 kB/s | 49 kB 00:00 2024-03-16T10:53:47.918 INFO:teuthology.orchestra.run.smithi078.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 610 kB/s | 51 kB 00:00 2024-03-16T10:53:47.943 INFO:teuthology.orchestra.run.smithi078.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-16T10:53:47.960 INFO:teuthology.orchestra.run.smithi153.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 283 kB/s | 16 kB 00:00 2024-03-16T10:53:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-16T10:53:47.994 INFO:teuthology.orchestra.run.smithi078.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-16T10:53:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-16T10:53:48.044 INFO:teuthology.orchestra.run.smithi078.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-16T10:53:48.095 INFO:teuthology.orchestra.run.smithi078.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-16T10:53:48.145 INFO:teuthology.orchestra.run.smithi078.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-16T10:53:48.185 INFO:teuthology.orchestra.run.smithi153.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.1 MB/s | 264 kB 00:00 2024-03-16T10:53:48.278 INFO:teuthology.orchestra.run.smithi153.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 5.2 MB/s | 7.4 MB 00:01 2024-03-16T10:53:48.344 INFO:teuthology.orchestra.run.smithi153.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 275 kB/s | 18 kB 00:00 2024-03-16T10:53:48.379 INFO:teuthology.orchestra.run.smithi078.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 9.8 MB/s | 2.3 MB 00:00 2024-03-16T10:53:48.411 INFO:teuthology.orchestra.run.smithi098.stdout:(23/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 8.7 MB/s | 48 MB 00:05 2024-03-16T10:53:48.471 INFO:teuthology.orchestra.run.smithi078.stdout:(153/174): librados2-19.0.0-2162.g3d78f5ba.el9. 7.5 MB/s | 3.4 MB 00:00 2024-03-16T10:53:48.478 INFO:teuthology.orchestra.run.smithi153.stdout:(38/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 2.5 MB/s | 748 kB 00:00 2024-03-16T10:53:48.487 INFO:teuthology.orchestra.run.smithi098.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 435 kB/s | 298 kB 00:00 2024-03-16T10:53:48.503 INFO:teuthology.orchestra.run.smithi153.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-16T10:53:48.504 INFO:teuthology.orchestra.run.smithi098.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.1 MB/s | 99 kB 00:00 2024-03-16T10:53:48.529 INFO:teuthology.orchestra.run.smithi078.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 731 kB/s | 42 kB 00:00 2024-03-16T10:53:48.545 INFO:teuthology.orchestra.run.smithi153.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 13 MB/s | 920 kB 00:00 2024-03-16T10:53:48.571 INFO:teuthology.orchestra.run.smithi153.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-16T10:53:48.579 INFO:teuthology.orchestra.run.smithi078.stdout:(155/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 5.9 MB/s | 3.1 MB 00:00 2024-03-16T10:53:48.596 INFO:teuthology.orchestra.run.smithi098.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 787 kB/s | 85 kB 00:00 2024-03-16T10:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-16T10:53:48.621 INFO:teuthology.orchestra.run.smithi153.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 639 kB/s | 32 kB 00:00 2024-03-16T10:53:48.621 INFO:teuthology.orchestra.run.smithi078.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 7.5 MB/s | 1.8 MB 00:00 2024-03-16T10:53:48.672 INFO:teuthology.orchestra.run.smithi078.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-16T10:53:48.687 INFO:teuthology.orchestra.run.smithi098.stdout:(27/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 3.6 MB/s | 5.1 MB 00:01 2024-03-16T10:53:48.696 INFO:teuthology.orchestra.run.smithi153.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-16T10:53:48.697 INFO:teuthology.orchestra.run.smithi078.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 486 kB/s | 36 kB 00:00 2024-03-16T10:53:48.721 INFO:teuthology.orchestra.run.smithi153.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 175 kB/s | 17 kB 00:00 2024-03-16T10:53:48.722 INFO:teuthology.orchestra.run.smithi078.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 502 kB/s | 71 kB 00:00 2024-03-16T10:53:48.738 INFO:teuthology.orchestra.run.smithi098.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 682 kB/s | 34 kB 00:00 2024-03-16T10:53:48.747 INFO:teuthology.orchestra.run.smithi153.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-16T10:53:48.763 INFO:teuthology.orchestra.run.smithi098.stdout:(29/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.0 MB/s | 170 kB 00:00 2024-03-16T10:53:48.772 INFO:teuthology.orchestra.run.smithi153.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-16T10:53:48.772 INFO:teuthology.orchestra.run.smithi078.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-16T10:53:48.789 INFO:teuthology.orchestra.run.smithi078.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 346 kB/s | 23 kB 00:00 2024-03-16T10:53:48.797 INFO:teuthology.orchestra.run.smithi098.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.3 MB/s | 139 kB 00:00 2024-03-16T10:53:48.797 INFO:teuthology.orchestra.run.smithi153.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-16T10:53:48.823 INFO:teuthology.orchestra.run.smithi153.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T10:53:48.823 INFO:teuthology.orchestra.run.smithi078.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-16T10:53:48.856 INFO:teuthology.orchestra.run.smithi153.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-16T10:53:48.865 INFO:teuthology.orchestra.run.smithi078.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-03-16T10:53:48.882 INFO:teuthology.orchestra.run.smithi153.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 804 kB/s | 46 kB 00:00 2024-03-16T10:53:48.889 INFO:teuthology.orchestra.run.smithi098.stdout:(31/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 8.0 MB/s | 3.1 MB 00:00 2024-03-16T10:53:48.906 INFO:teuthology.orchestra.run.smithi078.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-03-16T10:53:48.907 INFO:teuthology.orchestra.run.smithi153.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-16T10:53:48.932 INFO:teuthology.orchestra.run.smithi078.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-16T10:53:48.932 INFO:teuthology.orchestra.run.smithi153.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-16T10:53:48.957 INFO:teuthology.orchestra.run.smithi078.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 695 kB/s | 64 kB 00:00 2024-03-16T10:53:48.958 INFO:teuthology.orchestra.run.smithi153.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-16T10:53:48.964 INFO:teuthology.orchestra.run.smithi098.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 3.2 MB/s | 248 kB 00:00 2024-03-16T10:53:48.982 INFO:teuthology.orchestra.run.smithi078.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-03-16T10:53:48.983 INFO:teuthology.orchestra.run.smithi153.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-16T10:53:49.039 INFO:teuthology.orchestra.run.smithi098.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 658 kB/s | 49 kB 00:00 2024-03-16T10:53:49.066 INFO:teuthology.orchestra.run.smithi078.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-03-16T10:53:49.089 INFO:teuthology.orchestra.run.smithi098.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 329 kB/s | 16 kB 00:00 2024-03-16T10:53:49.100 INFO:teuthology.orchestra.run.smithi153.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-03-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi078.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi153.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-16T10:53:49.165 INFO:teuthology.orchestra.run.smithi098.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.4 MB/s | 264 kB 00:00 2024-03-16T10:53:49.167 INFO:teuthology.orchestra.run.smithi153.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-16T10:53:49.183 INFO:teuthology.orchestra.run.smithi078.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-16T10:53:49.192 INFO:teuthology.orchestra.run.smithi153.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-16T10:53:49.217 INFO:teuthology.orchestra.run.smithi078.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 3.0 MB/s | 465 kB 00:00 2024-03-16T10:53:49.242 INFO:teuthology.orchestra.run.smithi078.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 286 kB/s | 31 kB 00:00 2024-03-16T10:53:49.248 INFO:teuthology.orchestra.run.smithi098.stdout:(36/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 3.7 MB/s | 1.8 MB 00:00 2024-03-16T10:53:49.315 INFO:teuthology.orchestra.run.smithi098.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.9 MB/s | 748 kB 00:00 2024-03-16T10:53:49.317 INFO:teuthology.orchestra.run.smithi078.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-03-16T10:53:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 218 kB/s | 18 kB 00:00 2024-03-16T10:53:49.334 INFO:teuthology.orchestra.run.smithi153.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-16T10:53:49.342 INFO:teuthology.orchestra.run.smithi078.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 593 kB/s | 74 kB 00:00 2024-03-16T10:53:49.347 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:53:49.347 INFO:teuthology.orchestra.run.smithi078.stdout:Total 14 MB/s | 224 MB 00:15 2024-03-16T10:53:49.368 INFO:teuthology.orchestra.run.smithi153.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-16T10:53:49.393 INFO:teuthology.orchestra.run.smithi153.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-16T10:53:49.566 INFO:teuthology.orchestra.run.smithi098.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-03-16T10:53:49.599 INFO:teuthology.orchestra.run.smithi098.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-16T10:53:49.625 INFO:teuthology.orchestra.run.smithi098.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-16T10:53:49.650 INFO:teuthology.orchestra.run.smithi098.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-16T10:53:49.667 INFO:teuthology.orchestra.run.smithi098.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 765 kB/s | 32 kB 00:00 2024-03-16T10:53:49.751 INFO:teuthology.orchestra.run.smithi098.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-16T10:53:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 171 kB/s | 17 kB 00:00 2024-03-16T10:53:49.785 INFO:teuthology.orchestra.run.smithi098.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 300 kB/s | 10 kB 00:00 2024-03-16T10:53:49.809 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-16T10:53:49.810 INFO:teuthology.orchestra.run.smithi098.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-16T10:53:49.835 INFO:teuthology.orchestra.run.smithi098.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-16T10:53:49.852 INFO:teuthology.orchestra.run.smithi153.stdout:(63/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 6.0 MB/s | 48 MB 00:07 2024-03-16T10:53:49.860 INFO:teuthology.orchestra.run.smithi098.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T10:53:49.886 INFO:teuthology.orchestra.run.smithi098.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-16T10:53:49.919 INFO:teuthology.orchestra.run.smithi098.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 798 kB/s | 46 kB 00:00 2024-03-16T10:53:49.936 INFO:teuthology.orchestra.run.smithi153.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2024-03-16T10:53:49.945 INFO:teuthology.orchestra.run.smithi098.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-16T10:53:49.953 INFO:teuthology.orchestra.run.smithi153.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 44 kB/s | 24 kB 00:00 2024-03-16T10:53:49.969 INFO:teuthology.orchestra.run.smithi153.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 158 kB/s | 18 kB 00:00 2024-03-16T10:53:49.970 INFO:teuthology.orchestra.run.smithi098.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-16T10:53:49.986 INFO:teuthology.orchestra.run.smithi153.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 953 kB/s | 32 kB 00:00 2024-03-16T10:53:49.995 INFO:teuthology.orchestra.run.smithi098.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-16T10:53:50.012 INFO:teuthology.orchestra.run.smithi153.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-16T10:53:50.021 INFO:teuthology.orchestra.run.smithi098.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 682 kB/s | 34 kB 00:00 2024-03-16T10:53:50.113 INFO:teuthology.orchestra.run.smithi098.stdout:(56/174): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-03-16T10:53:50.118 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-16T10:53:50.118 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-16T10:53:50.162 INFO:teuthology.orchestra.run.smithi153.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-16T10:53:50.187 INFO:teuthology.orchestra.run.smithi153.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-16T10:53:50.212 INFO:teuthology.orchestra.run.smithi153.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 883 kB/s | 174 kB 00:00 2024-03-16T10:53:50.213 INFO:teuthology.orchestra.run.smithi098.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-16T10:53:50.237 INFO:teuthology.orchestra.run.smithi153.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 923 kB/s | 69 kB 00:00 2024-03-16T10:53:50.255 INFO:teuthology.orchestra.run.smithi098.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 6.5 MB/s | 1.5 MB 00:00 2024-03-16T10:53:50.263 INFO:teuthology.orchestra.run.smithi153.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 827 kB/s | 41 kB 00:00 2024-03-16T10:53:50.280 INFO:teuthology.orchestra.run.smithi098.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.8 MB/s | 811 kB 00:00 2024-03-16T10:53:50.288 INFO:teuthology.orchestra.run.smithi153.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-16T10:53:50.305 INFO:teuthology.orchestra.run.smithi098.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-16T10:53:50.313 INFO:teuthology.orchestra.run.smithi153.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-16T10:53:50.356 INFO:teuthology.orchestra.run.smithi098.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-16T10:53:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 635 kB/s | 47 kB 00:00 2024-03-16T10:53:50.389 INFO:teuthology.orchestra.run.smithi153.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 211 kB 00:00 2024-03-16T10:53:50.414 INFO:teuthology.orchestra.run.smithi153.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 427 kB/s | 43 kB 00:00 2024-03-16T10:53:50.464 INFO:teuthology.orchestra.run.smithi098.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-16T10:53:50.490 INFO:teuthology.orchestra.run.smithi098.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 424 kB/s | 78 kB 00:00 2024-03-16T10:53:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-16T10:53:50.639 INFO:teuthology.orchestra.run.smithi153.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 174 kB/s | 39 kB 00:00 2024-03-16T10:53:50.640 INFO:teuthology.orchestra.run.smithi098.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 141 kB/s | 24 kB 00:00 2024-03-16T10:53:50.657 INFO:teuthology.orchestra.run.smithi098.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 110 kB/s | 18 kB 00:00 2024-03-16T10:53:50.664 INFO:teuthology.orchestra.run.smithi153.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 166 kB/s | 41 kB 00:00 2024-03-16T10:53:50.682 INFO:teuthology.orchestra.run.smithi098.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-16T10:53:50.699 INFO:teuthology.orchestra.run.smithi098.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 757 kB/s | 32 kB 00:00 2024-03-16T10:53:50.816 INFO:teuthology.orchestra.run.smithi098.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-16T10:53:50.841 INFO:teuthology.orchestra.run.smithi098.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-16T10:53:50.848 INFO:teuthology.orchestra.run.smithi153.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-16T10:53:50.867 INFO:teuthology.orchestra.run.smithi098.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-16T10:53:50.873 INFO:teuthology.orchestra.run.smithi153.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-16T10:53:50.892 INFO:teuthology.orchestra.run.smithi098.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 918 kB/s | 69 kB 00:00 2024-03-16T10:53:50.898 INFO:teuthology.orchestra.run.smithi153.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 244 kB/s | 56 kB 00:00 2024-03-16T10:53:50.917 INFO:teuthology.orchestra.run.smithi098.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-16T10:53:50.924 INFO:teuthology.orchestra.run.smithi153.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 842 kB/s | 63 kB 00:00 2024-03-16T10:53:50.934 INFO:teuthology.orchestra.run.smithi098.stdout:(74/174): python3-prettytable-0.7.2-27.el9.noar 613 kB/s | 41 kB 00:00 2024-03-16T10:53:50.949 INFO:teuthology.orchestra.run.smithi153.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-16T10:53:50.959 INFO:teuthology.orchestra.run.smithi098.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-16T10:53:50.974 INFO:teuthology.orchestra.run.smithi153.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-16T10:53:50.985 INFO:teuthology.orchestra.run.smithi098.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-16T10:53:50.999 INFO:teuthology.orchestra.run.smithi153.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-16T10:53:51.001 INFO:teuthology.orchestra.run.smithi098.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-16T10:53:51.018 INFO:teuthology.orchestra.run.smithi098.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 736 kB/s | 43 kB 00:00 2024-03-16T10:53:51.025 INFO:teuthology.orchestra.run.smithi153.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-16T10:53:51.058 INFO:teuthology.orchestra.run.smithi153.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-16T10:53:51.084 INFO:teuthology.orchestra.run.smithi153.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-16T10:53:51.109 INFO:teuthology.orchestra.run.smithi153.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-16T10:53:51.202 INFO:teuthology.orchestra.run.smithi098.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 213 kB/s | 39 kB 00:00 2024-03-16T10:53:51.219 INFO:teuthology.orchestra.run.smithi098.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 206 kB/s | 41 kB 00:00 2024-03-16T10:53:51.343 INFO:teuthology.orchestra.run.smithi153.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 725 kB/s | 187 kB 00:00 2024-03-16T10:53:51.368 INFO:teuthology.orchestra.run.smithi153.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 755 kB/s | 195 kB 00:00 2024-03-16T10:53:51.401 INFO:teuthology.orchestra.run.smithi153.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-16T10:53:51.403 INFO:teuthology.orchestra.run.smithi098.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-16T10:53:51.427 INFO:teuthology.orchestra.run.smithi153.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-16T10:53:51.428 INFO:teuthology.orchestra.run.smithi098.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-16T10:53:51.445 INFO:teuthology.orchestra.run.smithi098.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 252 kB/s | 56 kB 00:00 2024-03-16T10:53:51.462 INFO:teuthology.orchestra.run.smithi098.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-16T10:53:51.479 INFO:teuthology.orchestra.run.smithi098.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-16T10:53:51.485 INFO:teuthology.orchestra.run.smithi153.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 183 kB/s | 21 kB 00:00 2024-03-16T10:53:51.504 INFO:teuthology.orchestra.run.smithi098.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-16T10:53:51.511 INFO:teuthology.orchestra.run.smithi153.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-03-16T10:53:51.529 INFO:teuthology.orchestra.run.smithi098.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-16T10:53:51.554 INFO:teuthology.orchestra.run.smithi098.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-16T10:53:51.588 INFO:teuthology.orchestra.run.smithi098.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-16T10:53:51.613 INFO:teuthology.orchestra.run.smithi098.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-16T10:53:51.627 INFO:teuthology.orchestra.run.smithi153.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-16T10:53:51.639 INFO:teuthology.orchestra.run.smithi098.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-16T10:53:51.653 INFO:teuthology.orchestra.run.smithi153.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-03-16T10:53:51.678 INFO:teuthology.orchestra.run.smithi153.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-16T10:53:51.695 INFO:teuthology.orchestra.run.smithi153.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 839 kB/s | 35 kB 00:00 2024-03-16T10:53:51.720 INFO:teuthology.orchestra.run.smithi153.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-16T10:53:51.745 INFO:teuthology.orchestra.run.smithi153.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-16T10:53:51.829 INFO:teuthology.orchestra.run.smithi153.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 10 MB/s | 4.1 MB 00:00 2024-03-16T10:53:51.921 INFO:teuthology.orchestra.run.smithi153.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-16T10:53:52.030 INFO:teuthology.orchestra.run.smithi153.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.7 MB/s | 548 kB 00:00 2024-03-16T10:53:52.063 INFO:teuthology.orchestra.run.smithi153.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 588 kB/s | 19 kB 00:00 2024-03-16T10:53:52.114 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-16T10:53:52.114 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-16T10:53:52.122 INFO:teuthology.orchestra.run.smithi153.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-03-16T10:53:52.139 INFO:teuthology.orchestra.run.smithi098.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 356 kB/s | 187 kB 00:00 2024-03-16T10:53:52.147 INFO:teuthology.orchestra.run.smithi153.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-16T10:53:52.181 INFO:teuthology.orchestra.run.smithi098.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 360 kB/s | 195 kB 00:00 2024-03-16T10:53:52.239 INFO:teuthology.orchestra.run.smithi153.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-03-16T10:53:52.248 INFO:teuthology.orchestra.run.smithi098.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-16T10:53:52.264 INFO:teuthology.orchestra.run.smithi153.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-03-16T10:53:52.273 INFO:teuthology.orchestra.run.smithi098.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-16T10:53:52.290 INFO:teuthology.orchestra.run.smithi153.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 342 kB/s | 49 kB 00:00 2024-03-16T10:53:52.298 INFO:teuthology.orchestra.run.smithi098.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-03-16T10:53:52.315 INFO:teuthology.orchestra.run.smithi153.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 895 kB/s | 67 kB 00:00 2024-03-16T10:53:52.332 INFO:teuthology.orchestra.run.smithi098.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-16T10:53:52.340 INFO:teuthology.orchestra.run.smithi153.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-16T10:53:52.365 INFO:teuthology.orchestra.run.smithi153.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 11 MB/s | 548 kB 00:00 2024-03-16T10:53:52.374 INFO:teuthology.orchestra.run.smithi098.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-16T10:53:52.391 INFO:teuthology.orchestra.run.smithi153.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 866 kB/s | 43 kB 00:00 2024-03-16T10:53:52.416 INFO:teuthology.orchestra.run.smithi153.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-03-16T10:53:52.416 INFO:teuthology.orchestra.run.smithi098.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 216 kB 00:00 2024-03-16T10:53:52.441 INFO:teuthology.orchestra.run.smithi153.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-16T10:53:52.458 INFO:teuthology.orchestra.run.smithi098.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 840 kB/s | 35 kB 00:00 2024-03-16T10:53:52.466 INFO:teuthology.orchestra.run.smithi153.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-16T10:53:52.483 INFO:teuthology.orchestra.run.smithi098.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.6 MB/s | 248 kB 00:00 2024-03-16T10:53:52.500 INFO:teuthology.orchestra.run.smithi098.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-16T10:53:52.500 INFO:teuthology.orchestra.run.smithi153.stdout:(120/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi153.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-03-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi098.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-16T10:53:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 530 kB/s | 22 kB 00:00 2024-03-16T10:53:52.551 INFO:teuthology.orchestra.run.smithi153.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 705 kB/s | 35 kB 00:00 2024-03-16T10:53:52.576 INFO:teuthology.orchestra.run.smithi153.stdout:(123/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-16T10:53:52.601 INFO:teuthology.orchestra.run.smithi153.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-16T10:53:52.618 INFO:teuthology.orchestra.run.smithi098.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 12 MB/s | 4.1 MB 00:00 2024-03-16T10:53:52.668 INFO:teuthology.orchestra.run.smithi153.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-03-16T10:53:52.676 INFO:teuthology.orchestra.run.smithi098.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-03-16T10:53:52.694 INFO:teuthology.orchestra.run.smithi153.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-16T10:53:52.719 INFO:teuthology.orchestra.run.smithi153.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-16T10:53:52.736 INFO:teuthology.orchestra.run.smithi153.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 263 kB/s | 18 kB 00:00 2024-03-16T10:53:52.753 INFO:teuthology.orchestra.run.smithi153.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-03-16T10:53:52.770 INFO:teuthology.orchestra.run.smithi153.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 582 kB/s | 19 kB 00:00 2024-03-16T10:53:52.786 INFO:teuthology.orchestra.run.smithi153.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-03-16T10:53:52.803 INFO:teuthology.orchestra.run.smithi153.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-03-16T10:53:52.845 INFO:teuthology.orchestra.run.smithi153.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-16T10:53:52.862 INFO:teuthology.orchestra.run.smithi153.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-03-16T10:53:52.888 INFO:teuthology.orchestra.run.smithi153.stdout:(135/174): python3-more-itertools-8.12.0-2.el9. 783 kB/s | 79 kB 00:00 2024-03-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi098.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-16T10:53:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 392 kB/s | 16 kB 00:00 2024-03-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi153.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-16T10:53:52.947 INFO:teuthology.orchestra.run.smithi153.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 674 kB/s | 28 kB 00:00 2024-03-16T10:53:52.964 INFO:teuthology.orchestra.run.smithi153.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-16T10:53:52.981 INFO:teuthology.orchestra.run.smithi153.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-03-16T10:53:53.006 INFO:teuthology.orchestra.run.smithi153.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-16T10:53:53.023 INFO:teuthology.orchestra.run.smithi153.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 853 kB/s | 36 kB 00:00 2024-03-16T10:53:53.040 INFO:teuthology.orchestra.run.smithi153.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 779 kB/s | 59 kB 00:00 2024-03-16T10:53:53.057 INFO:teuthology.orchestra.run.smithi153.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-16T10:53:53.066 INFO:teuthology.orchestra.run.smithi098.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-16T10:53:53.082 INFO:teuthology.orchestra.run.smithi153.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi098.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 688 kB/s | 309 kB 00:00 2024-03-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi153.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-16T10:53:53.132 INFO:teuthology.orchestra.run.smithi153.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-03-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi098.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 373 kB/s | 26 kB 00:00 2024-03-16T10:53:53.169 INFO:teuthology.orchestra.run.smithi098.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 642 kB/s | 49 kB 00:00 2024-03-16T10:53:53.183 INFO:teuthology.orchestra.run.smithi153.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-16T10:53:53.208 INFO:teuthology.orchestra.run.smithi153.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-16T10:53:53.220 INFO:teuthology.orchestra.run.smithi098.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 805 kB/s | 67 kB 00:00 2024-03-16T10:53:53.253 INFO:teuthology.orchestra.run.smithi098.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi153.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-16T10:53:53.370 INFO:teuthology.orchestra.run.smithi098.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2024-03-16T10:53:53.396 INFO:teuthology.orchestra.run.smithi098.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-16T10:53:53.425 INFO:teuthology.orchestra.run.smithi153.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-16T10:53:53.429 INFO:teuthology.orchestra.run.smithi098.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 3.0 MB/s | 548 kB 00:00 2024-03-16T10:53:53.454 INFO:teuthology.orchestra.run.smithi098.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 518 kB/s | 43 kB 00:00 2024-03-16T10:53:53.480 INFO:teuthology.orchestra.run.smithi098.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 384 kB/s | 32 kB 00:00 2024-03-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi098.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-03-16T10:53:53.530 INFO:teuthology.orchestra.run.smithi098.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-16T10:53:53.556 INFO:teuthology.orchestra.run.smithi098.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-16T10:53:53.598 INFO:teuthology.orchestra.run.smithi098.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-16T10:53:53.623 INFO:teuthology.orchestra.run.smithi098.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-03-16T10:53:53.625 INFO:teuthology.orchestra.run.smithi153.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 7.7 MB/s | 3.4 MB 00:00 2024-03-16T10:53:53.648 INFO:teuthology.orchestra.run.smithi098.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.1 MB/s | 203 kB 00:00 2024-03-16T10:53:53.690 INFO:teuthology.orchestra.run.smithi098.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-16T10:53:53.715 INFO:teuthology.orchestra.run.smithi098.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 159 kB/s | 11 kB 00:00 2024-03-16T10:53:53.774 INFO:teuthology.orchestra.run.smithi098.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-16T10:53:53.799 INFO:teuthology.orchestra.run.smithi098.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-03-16T10:53:53.825 INFO:teuthology.orchestra.run.smithi098.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 212 kB/s | 23 kB 00:00 2024-03-16T10:53:53.850 INFO:teuthology.orchestra.run.smithi098.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-16T10:53:53.859 INFO:teuthology.orchestra.run.smithi153.stdout:(153/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-03-16T10:53:53.960 INFO:teuthology.orchestra.run.smithi153.stdout:(154/174): NetworkManager-1.46.0-1.el9.x86_64.r 4.3 MB/s | 2.3 MB 00:00 2024-03-16T10:53:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 338 kB/s | 42 kB 00:00 2024-03-16T10:53:54.009 INFO:teuthology.orchestra.run.smithi098.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 107 kB/s | 20 kB 00:00 2024-03-16T10:53:54.034 INFO:teuthology.orchestra.run.smithi098.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 252 kB/s | 46 kB 00:00 2024-03-16T10:53:54.044 INFO:teuthology.orchestra.run.smithi153.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-16T10:53:54.068 INFO:teuthology.orchestra.run.smithi098.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 4.2 MB/s | 1.0 MB 00:00 2024-03-16T10:53:54.093 INFO:teuthology.orchestra.run.smithi098.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 940 kB/s | 79 kB 00:00 2024-03-16T10:53:54.094 INFO:teuthology.orchestra.run.smithi153.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 658 kB/s | 71 kB 00:00 2024-03-16T10:53:54.118 INFO:teuthology.orchestra.run.smithi098.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 687 kB/s | 58 kB 00:00 2024-03-16T10:53:54.119 INFO:teuthology.orchestra.run.smithi153.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 484 kB/s | 36 kB 00:00 2024-03-16T10:53:54.143 INFO:teuthology.orchestra.run.smithi098.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-16T10:53:54.161 INFO:teuthology.orchestra.run.smithi153.stdout:(159/174): NetworkManager-libnm-1.46.0-1.el9.x8 3.4 MB/s | 1.8 MB 00:00 2024-03-16T10:53:54.169 INFO:teuthology.orchestra.run.smithi098.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-16T10:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-16T10:53:54.186 INFO:teuthology.orchestra.run.smithi153.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-16T10:53:54.211 INFO:teuthology.orchestra.run.smithi098.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 419 kB/s | 28 kB 00:00 2024-03-16T10:53:54.211 INFO:teuthology.orchestra.run.smithi153.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-16T10:53:54.228 INFO:teuthology.orchestra.run.smithi098.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 520 kB/s | 31 kB 00:00 2024-03-16T10:53:54.229 INFO:teuthology.orchestra.run.smithi153.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 343 kB/s | 23 kB 00:00 2024-03-16T10:53:54.253 INFO:teuthology.orchestra.run.smithi098.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-03-16T10:53:54.254 INFO:teuthology.orchestra.run.smithi153.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-16T10:53:54.270 INFO:teuthology.orchestra.run.smithi098.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-16T10:53:54.279 INFO:teuthology.orchestra.run.smithi153.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-16T10:53:54.287 INFO:teuthology.orchestra.run.smithi098.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 608 kB/s | 36 kB 00:00 2024-03-16T10:53:54.304 INFO:teuthology.orchestra.run.smithi153.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-16T10:53:54.329 INFO:teuthology.orchestra.run.smithi098.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-03-16T10:53:54.330 INFO:teuthology.orchestra.run.smithi153.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 846 kB/s | 64 kB 00:00 2024-03-16T10:53:54.355 INFO:teuthology.orchestra.run.smithi153.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-03-16T10:53:54.363 INFO:teuthology.orchestra.run.smithi098.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-16T10:53:54.388 INFO:teuthology.orchestra.run.smithi098.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 893 kB/s | 90 kB 00:00 2024-03-16T10:53:54.414 INFO:teuthology.orchestra.run.smithi098.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-16T10:53:54.439 INFO:teuthology.orchestra.run.smithi098.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-16T10:53:54.464 INFO:teuthology.orchestra.run.smithi098.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-16T10:53:54.514 INFO:teuthology.orchestra.run.smithi098.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-16T10:53:54.581 INFO:teuthology.orchestra.run.smithi098.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.9 MB/s | 321 kB 00:00 2024-03-16T10:53:54.588 INFO:teuthology.orchestra.run.smithi153.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.3 MB/s | 1.2 MB 00:00 2024-03-16T10:53:54.705 INFO:teuthology.orchestra.run.smithi153.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-03-16T10:53:54.764 INFO:teuthology.orchestra.run.smithi153.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 534 kB/s | 31 kB 00:00 2024-03-16T10:53:54.806 INFO:teuthology.orchestra.run.smithi153.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.1 MB/s | 2.3 MB 00:00 2024-03-16T10:53:54.848 INFO:teuthology.orchestra.run.smithi153.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.2 MB/s | 2.2 MB 00:00 2024-03-16T10:53:54.898 INFO:teuthology.orchestra.run.smithi153.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 811 kB/s | 74 kB 00:00 2024-03-16T10:53:54.907 INFO:teuthology.orchestra.run.smithi098.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 7.3 MB/s | 3.4 MB 00:00 2024-03-16T10:53:54.957 INFO:teuthology.orchestra.run.smithi098.stdout:(153/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 6.4 MB/s | 3.1 MB 00:00 2024-03-16T10:53:54.998 INFO:teuthology.orchestra.run.smithi153.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.6 MB/s | 1.1 MB 00:00 2024-03-16T10:53:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:53:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:Total 12 MB/s | 224 MB 00:19 2024-03-16T10:53:55.224 INFO:teuthology.orchestra.run.smithi098.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 159 kB/s | 42 kB 00:00 2024-03-16T10:53:55.374 INFO:teuthology.orchestra.run.smithi098.stdout:(155/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.9 MB/s | 2.3 MB 00:00 2024-03-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi098.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 714 kB/s | 71 kB 00:00 2024-03-16T10:53:55.499 INFO:teuthology.orchestra.run.smithi098.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 900 kB/s | 248 kB 00:00 2024-03-16T10:53:55.600 INFO:teuthology.orchestra.run.smithi098.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 291 kB/s | 36 kB 00:00 2024-03-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi098.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-16T10:53:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-03-16T10:53:55.726 INFO:teuthology.orchestra.run.smithi098.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-16T10:53:55.768 INFO:teuthology.orchestra.run.smithi098.stdout:(162/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.1 MB/s | 1.8 MB 00:00 2024-03-16T10:53:55.781 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-16T10:53:55.781 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-16T10:53:55.810 INFO:teuthology.orchestra.run.smithi098.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 994 kB/s | 108 kB 00:00 2024-03-16T10:53:55.835 INFO:teuthology.orchestra.run.smithi098.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-16T10:53:55.877 INFO:teuthology.orchestra.run.smithi098.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-16T10:53:55.910 INFO:teuthology.orchestra.run.smithi098.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 635 kB/s | 64 kB 00:00 2024-03-16T10:53:55.936 INFO:teuthology.orchestra.run.smithi098.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 285 kB/s | 29 kB 00:00 2024-03-16T10:53:56.087 INFO:teuthology.orchestra.run.smithi098.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.8 MB/s | 1.2 MB 00:00 2024-03-16T10:53:56.220 INFO:teuthology.orchestra.run.smithi098.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.0 MB/s | 2.2 MB 00:00 2024-03-16T10:53:56.245 INFO:teuthology.orchestra.run.smithi098.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.9 MB/s | 465 kB 00:00 2024-03-16T10:53:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-16T10:53:56.371 INFO:teuthology.orchestra.run.smithi098.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-16T10:53:56.396 INFO:teuthology.orchestra.run.smithi098.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 74 kB 00:00 2024-03-16T10:53:56.438 INFO:teuthology.orchestra.run.smithi098.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.6 MB/s | 2.3 MB 00:00 2024-03-16T10:53:56.445 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:53:56.445 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 224 MB 00:21 2024-03-16T10:53:56.910 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-16T10:53:57.220 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-16T10:53:57.220 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-16T10:53:57.772 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-16T10:53:57.772 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-16T10:53:59.196 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-16T10:53:59.304 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-16T10:53:59.305 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-16T10:53:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T10:53:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T10:53:59.395 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:53:59.404 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:53:59.444 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T10:53:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T10:53:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T10:53:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T10:53:59.636 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T10:53:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T10:54:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:00.300 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:54:00.333 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T10:54:00.516 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T10:54:00.561 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T10:54:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:00.676 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:00.698 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:54:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:54:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T10:54:00.825 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:54:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:54:01.033 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:54:01.065 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:54:01.096 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T10:54:01.266 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T10:54:01.381 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:54:01.425 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:54:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:01.584 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:01.799 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:01.907 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T10:54:02.008 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T10:54:02.039 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T10:54:02.066 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T10:54:02.101 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T10:54:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T10:54:02.228 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T10:54:02.267 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T10:54:02.330 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T10:54:02.366 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T10:54:02.409 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T10:54:02.442 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T10:54:02.470 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T10:54:02.539 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T10:54:02.574 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T10:54:02.624 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T10:54:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T10:54:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T10:54:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T10:54:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T10:54:03.021 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T10:54:03.541 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T10:54:03.643 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T10:54:03.732 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T10:54:03.772 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:54:03.782 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:54:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T10:54:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T10:54:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T10:54:03.929 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T10:54:04.210 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T10:54:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T10:54:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T10:54:04.524 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T10:54:04.668 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T10:54:04.711 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T10:54:04.766 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T10:54:04.829 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T10:54:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T10:54:04.898 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T10:54:04.901 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-16T10:54:04.945 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T10:54:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T10:54:05.010 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T10:54:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T10:54:05.037 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T10:54:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T10:54:05.075 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:54:05.085 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:54:05.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T10:54:05.186 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T10:54:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T10:54:05.220 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T10:54:05.246 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T10:54:05.300 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T10:54:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T10:54:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T10:54:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T10:54:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T10:54:05.593 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T10:54:05.624 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T10:54:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T10:54:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T10:54:05.679 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T10:54:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T10:54:05.782 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:05.813 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T10:54:05.905 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:05.939 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:54:05.989 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T10:54:06.000 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T10:54:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T10:54:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T10:54:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T10:54:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T10:54:06.206 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T10:54:06.216 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-16T10:54:06.235 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T10:54:06.328 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T10:54:06.330 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T10:54:06.352 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:54:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T10:54:06.410 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:54:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:54:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:54:06.448 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T10:54:06.471 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:54:06.487 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:54:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T10:54:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T10:54:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T10:54:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T10:54:06.626 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T10:54:06.631 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:54:06.662 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:54:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T10:54:06.685 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T10:54:06.701 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T10:54:06.703 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T10:54:06.728 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T10:54:06.741 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T10:54:06.753 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T10:54:06.765 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T10:54:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T10:54:06.802 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T10:54:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T10:54:06.890 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T10:54:06.905 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T10:54:06.939 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T10:54:06.980 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T10:54:07.015 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T10:54:07.038 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:54:07.082 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:54:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T10:54:07.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T10:54:07.185 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T10:54:07.200 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:07.202 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T10:54:07.258 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T10:54:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T10:54:07.331 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:07.360 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:07.404 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T10:54:07.451 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:07.470 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:07.478 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T10:54:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:54:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T10:54:07.584 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T10:54:07.676 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T10:54:07.705 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T10:54:07.715 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T10:54:07.732 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T10:54:07.771 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T10:54:07.776 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T10:54:07.816 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:07.849 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T10:54:07.911 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T10:54:07.918 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:54:07.951 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T10:54:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:54:08.016 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T10:54:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T10:54:08.050 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T10:54:08.060 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:54:08.075 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:54:08.095 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T10:54:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T10:54:08.155 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T10:54:08.203 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T10:54:08.223 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T10:54:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T10:54:08.259 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T10:54:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:54:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T10:54:08.308 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:54:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T10:54:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T10:54:08.340 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T10:54:08.348 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T10:54:08.394 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T10:54:08.408 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T10:54:08.427 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T10:54:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T10:54:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T10:54:08.536 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T10:54:08.604 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T10:54:08.641 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T10:54:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T10:54:08.673 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:54:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T10:54:08.723 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T10:54:08.733 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:54:08.855 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:54:09.209 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T10:54:09.245 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T10:54:09.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T10:54:09.335 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T10:54:09.340 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T10:54:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T10:54:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T10:54:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T10:54:09.413 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T10:54:09.463 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:54:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T10:54:09.471 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:54:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T10:54:09.514 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T10:54:09.530 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T10:54:09.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T10:54:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T10:54:09.614 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T10:54:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T10:54:09.646 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T10:54:09.667 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T10:54:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T10:54:09.744 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T10:54:09.771 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T10:54:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T10:54:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T10:54:09.903 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T10:54:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T10:54:09.949 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T10:54:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T10:54:10.063 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T10:54:10.093 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T10:54:10.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T10:54:10.180 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T10:54:10.224 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T10:54:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T10:54:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T10:54:10.366 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T10:54:10.410 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T10:54:10.464 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T10:54:10.537 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T10:54:10.654 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T10:54:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T10:54:10.768 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T10:54:10.789 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T10:54:10.798 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T10:54:10.829 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T10:54:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T10:54:10.838 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T10:54:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T10:54:10.872 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T10:54:10.910 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T10:54:10.915 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T10:54:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T10:54:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T10:54:11.031 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:54:11.040 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:54:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T10:54:11.090 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T10:54:11.128 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T10:54:11.136 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T10:54:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T10:54:11.167 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T10:54:11.169 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T10:54:11.195 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:54:11.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T10:54:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:54:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T10:54:11.302 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T10:54:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T10:54:11.342 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T10:54:11.374 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T10:54:11.483 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T10:54:11.510 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T10:54:11.640 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T10:54:11.705 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T10:54:11.739 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T10:54:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T10:54:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T10:54:11.796 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T10:54:11.830 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T10:54:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:54:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T10:54:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:54:11.895 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T10:54:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T10:54:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T10:54:12.021 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T10:54:12.076 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T10:54:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T10:54:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T10:54:12.141 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T10:54:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T10:54:12.208 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T10:54:12.255 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T10:54:12.263 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T10:54:12.291 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T10:54:12.301 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T10:54:12.333 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T10:54:12.334 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T10:54:12.364 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T10:54:12.377 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T10:54:12.392 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T10:54:12.426 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T10:54:12.466 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T10:54:12.497 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T10:54:12.518 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T10:54:12.527 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T10:54:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T10:54:12.596 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T10:54:12.602 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T10:54:12.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T10:54:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T10:54:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T10:54:12.729 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T10:54:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T10:54:12.763 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T10:54:12.771 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T10:54:12.811 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T10:54:12.836 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T10:54:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T10:54:12.861 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T10:54:12.931 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T10:54:12.942 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T10:54:12.986 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T10:54:13.017 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T10:54:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T10:54:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T10:54:13.161 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T10:54:13.169 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T10:54:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T10:54:13.218 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T10:54:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T10:54:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T10:54:13.514 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T10:54:13.574 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T10:54:13.703 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T10:54:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T10:54:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T10:54:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T10:54:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T10:54:13.890 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T10:54:13.929 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T10:54:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T10:54:13.960 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T10:54:13.983 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T10:54:13.990 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T10:54:14.016 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T10:54:14.064 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T10:54:14.066 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T10:54:14.101 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T10:54:14.134 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T10:54:14.159 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T10:54:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T10:54:14.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T10:54:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T10:54:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T10:54:14.377 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T10:54:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T10:54:14.406 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T10:54:14.432 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T10:54:14.464 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T10:54:14.484 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T10:54:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T10:54:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T10:54:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T10:54:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:54:14.901 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:54:14.901 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T10:54:14.901 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T10:54:14.901 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:54:15.121 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T10:54:15.236 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T10:54:15.393 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T10:54:15.455 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T10:54:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T10:54:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T10:54:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T10:54:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T10:54:15.840 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T10:54:15.875 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T10:54:15.952 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T10:54:16.455 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T10:54:16.539 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T10:54:16.571 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T10:54:16.598 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T10:54:16.601 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T10:54:16.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T10:54:16.705 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T10:54:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T10:54:16.984 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:54:17.066 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:54:17.097 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T10:54:17.134 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T10:54:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T10:54:17.629 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:54:17.692 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:54:17.722 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T10:54:17.911 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T10:54:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T10:54:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T10:54:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T10:54:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T10:54:18.403 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T10:54:18.438 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:54:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:54:18.551 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T10:54:18.588 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T10:54:19.048 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T10:54:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:54:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:54:19.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T10:54:20.428 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:20.460 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:20.999 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:21.030 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:54:21.113 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:54:21.113 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T10:54:21.113 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T10:54:21.113 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:21.156 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:54:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:54:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:54:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:54:22.124 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T10:54:22.124 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T10:54:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:54:26.867 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:54:26.867 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-03-16T10:54:26.867 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-03-16T10:54:26.867 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-03-16T10:54:26.867 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-03-16T10:54:26.867 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-03-16T10:54:26.868 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /root 2024-03-16T10:54:26.868 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-03-16T10:54:26.868 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:27.030 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:54:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:54:27.046 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:27.046 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T10:54:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:54:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:54:28.559 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:28.559 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T10:54:28.559 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:54:28.559 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:54:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:54:28.815 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:54:28.815 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:28.815 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T10:54:28.815 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:54:28.815 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:54:28.815 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:29.260 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:54:29.282 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:54:29.282 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:29.282 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T10:54:29.282 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:54:29.283 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:54:29.283 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:29.378 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T10:54:29.763 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:54:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:54:29.825 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T10:54:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T10:54:29.909 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:54:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:54:30.175 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:54:31.134 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:54:31.159 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:54:31.259 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:54:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T10:54:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:54:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:54:31.484 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:31.484 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T10:54:31.484 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:54:31.484 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:54:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:31.529 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:54:31.542 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:54:31.588 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T10:54:32.458 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T10:54:32.489 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:54:32.512 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:54:32.512 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:32.512 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T10:54:32.512 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:54:32.512 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:54:32.512 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-16T10:54:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:54:32.570 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:32.570 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T10:54:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:32.637 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:32.637 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:32.637 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T10:54:32.638 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:54:32.638 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:54:32.638 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:54:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:54:32.740 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:32.740 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T10:54:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:54:32.963 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:54:32.964 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:32.964 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T10:54:32.964 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:54:32.964 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:54:32.964 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-16T10:54:34.057 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:34.219 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:54:34.219 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:54:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:54:34.234 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:34.234 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T10:54:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:34.244 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:54:34.245 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:34.245 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T10:54:34.245 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:54:34.245 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:54:34.245 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:54:34.493 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:54:34.493 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:34.493 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T10:54:34.494 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:54:34.494 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:54:34.494 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:34.939 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:54:34.961 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:54:34.962 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:34.962 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T10:54:34.962 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:54:34.962 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:54:34.962 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:35.057 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T10:54:35.416 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:54:35.448 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:54:35.479 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T10:54:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T10:54:35.562 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:54:35.722 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:54:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:54:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T10:54:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:54:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:54:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:35.795 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:54:35.830 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:54:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:54:36.003 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:54:36.003 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:36.003 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T10:54:36.003 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:54:36.003 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:54:36.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:54:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:54:36.462 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:36.462 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T10:54:36.462 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:54:36.462 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:54:36.462 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T10:54:36.789 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:54:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:54:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:54:37.001 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:54:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:54:37.067 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T10:54:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T10:54:37.090 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:54:37.113 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:54:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T10:54:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:54:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:54:37.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:37.140 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T10:54:37.157 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:54:37.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:54:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:54:37.218 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T10:54:37.397 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:54:37.404 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:54:37.422 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:54:37.462 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T10:54:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T10:54:37.528 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T10:54:37.706 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T10:54:37.820 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T10:54:37.859 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T10:54:37.916 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T10:54:37.976 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T10:54:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T10:54:38.052 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T10:54:38.085 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T10:54:38.093 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T10:54:38.118 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:54:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T10:54:38.141 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:54:38.141 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:38.141 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T10:54:38.142 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:54:38.142 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:54:38.142 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T10:54:38.185 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:54:38.207 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:54:38.207 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:38.207 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T10:54:38.207 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:38.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T10:54:38.275 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:38.276 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:38.276 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T10:54:38.276 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:54:38.276 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:54:38.276 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:38.295 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T10:54:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:54:38.406 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:54:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:54:38.587 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:54:38.609 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:54:38.610 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:38.610 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T10:54:38.610 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:54:38.610 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:54:38.610 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:38.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T10:54:38.708 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:54:38.731 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:54:38.731 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:38.731 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T10:54:38.731 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:54:38.731 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:54:38.731 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:38.776 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:54:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:54:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T10:54:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T10:54:39.714 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T10:54:39.745 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:54:39.767 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:54:39.767 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:39.767 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T10:54:39.767 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:54:39.767 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:54:39.767 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T10:54:39.812 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:54:39.833 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:54:39.833 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:39.833 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T10:54:39.834 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T10:54:39.881 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:39.903 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:39.903 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:39.903 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T10:54:39.903 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:54:39.904 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:54:39.904 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T10:54:39.967 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T10:54:39.992 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T10:54:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T10:54:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T10:54:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:54:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:54:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T10:54:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:54:40.207 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:54:40.229 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:54:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:54:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T10:54:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:54:40.230 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:54:40.230 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:40.246 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:54:40.272 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T10:54:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T10:54:40.323 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T10:54:40.348 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T10:54:40.382 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T10:54:40.428 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T10:54:40.460 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T10:54:40.485 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T10:54:40.511 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T10:54:40.536 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:54:40.562 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T10:54:40.568 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:40.568 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:40.748 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:40.748 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:54:40.833 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:54:40.858 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:54:40.883 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T10:54:40.916 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T10:54:40.942 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T10:54:41.008 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:54:43.050 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:54:43.093 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T10:54:43.121 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T10:54:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T10:54:43.328 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T10:54:43.447 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T10:54:43.487 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T10:54:43.540 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T10:54:43.588 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T10:54:43.630 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T10:54:43.663 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T10:54:43.703 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T10:54:43.733 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T10:54:43.825 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T10:54:43.897 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T10:54:43.932 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T10:54:44.478 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:44.514 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:44.535 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:44.670 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:54:44.728 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T10:54:44.758 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T10:54:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T10:54:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T10:54:45.099 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T10:54:45.142 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T10:54:45.187 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T10:54:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T10:54:45.278 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T10:54:45.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T10:54:45.350 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T10:54:45.352 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T10:54:45.380 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T10:54:45.434 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T10:54:45.454 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T10:54:45.475 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T10:54:45.523 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T10:54:45.556 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T10:54:45.562 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T10:54:45.588 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T10:54:45.613 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T10:54:45.638 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T10:54:45.673 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T10:54:45.673 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:54:45.752 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:54:45.782 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T10:54:45.782 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:54:45.858 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:54:45.884 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T10:54:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T10:54:45.943 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T10:54:45.968 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T10:54:46.005 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T10:54:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:54:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T10:54:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T10:54:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T10:54:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T10:54:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T10:54:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T10:54:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T10:54:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T10:54:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T10:54:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T10:54:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T10:54:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T10:54:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:54:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T10:54:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T10:54:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T10:54:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T10:54:46.041 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T10:54:46.072 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T10:54:46.098 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T10:54:46.123 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T10:54:46.148 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:54:46.150 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:46.165 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T10:54:46.171 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:46.172 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:46.204 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:54:46.351 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:46.351 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:54:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:54:46.450 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:54:46.476 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T10:54:46.509 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T10:54:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T10:54:46.600 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:54:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T10:54:47.123 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T10:54:47.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T10:54:47.255 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T10:54:47.285 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T10:54:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T10:54:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T10:54:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T10:54:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:54:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:54:47.488 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T10:54:47.488 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:54:47.572 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:54:47.598 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T10:54:47.632 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T10:54:47.657 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T10:54:47.682 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T10:54:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T10:54:47.764 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T10:54:47.795 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T10:54:47.821 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T10:54:47.846 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T10:54:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:54:47.897 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T10:54:47.903 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:47.903 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:48.108 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:54:48.108 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:54:48.174 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:54:48.199 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:54:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T10:54:48.265 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T10:54:48.292 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T10:54:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:54:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T10:54:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:49.225 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-03-16T10:54:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T10:54:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T10:54:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-16.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T10:54:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T10:54:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T10:54:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T10:54:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T10:54:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T10:54:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T10:54:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T10:54:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T10:54:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T10:54:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T10:54:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T10:54:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T10:54:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:54:49.238 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-16T10:54:49.712 DEBUG:teuthology.parallel:result is None 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T10:54:51.546 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T10:54:51.547 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T10:54:51.548 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T10:54:51.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T10:54:51.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T10:54:51.551 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T10:54:51.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T10:54:51.553 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T10:54:51.554 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T10:54:51.555 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T10:54:51.556 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T10:54:51.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T10:54:51.558 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T10:54:51.559 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T10:54:53.377 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:54:53.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T10:54:53.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T10:54:53.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T10:54:53.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T10:54:53.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T10:54:53.379 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T10:54:53.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T10:54:53.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T10:54:53.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T10:54:53.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T10:54:53.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T10:54:53.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T10:54:53.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T10:54:53.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T10:54:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T10:54:53.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T10:54:53.400 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T10:54:53.401 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T10:54:53.402 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T10:54:53.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T10:54:53.404 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T10:54:54.660 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T10:54:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T10:54:54.662 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.663 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:54.664 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T10:54:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T10:54:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T10:54:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T10:54:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T10:54:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T10:54:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T10:54:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T10:54:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T10:54:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T10:54:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:54:54.673 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-16T10:54:55.180 DEBUG:teuthology.parallel:result is None 2024-03-16T10:54:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T10:54:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T10:54:56.388 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T10:54:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T10:54:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T10:54:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T10:54:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T10:54:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T10:54:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T10:54:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T10:54:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T10:54:56.400 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-16T10:54:56.866 DEBUG:teuthology.parallel:result is None 2024-03-16T10:54:56.867 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:54:57.088 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T10:54:57.114 INFO:teuthology.orchestra.run.smithi078.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:54:57.114 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:54:57.114 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T10:54:57.116 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:54:57.253 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T10:54:57.277 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:54:57.278 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:54:57.278 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T10:54:57.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:54:57.494 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T10:54:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:54:57.519 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:54:57.519 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T10:54:57.520 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T10:54:57.521 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:54:57.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T10:54:57.555 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:54:57.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T10:54:57.589 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:54:57.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T10:54:57.623 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T10:54:57.624 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:54:57.624 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T10:54:57.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T10:54:57.724 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:54:57.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T10:54:57.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T10:54:57.825 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:54:57.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T10:54:57.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T10:54:57.927 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T10:54:57.927 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:54:57.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T10:54:57.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T10:54:58.025 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:54:58.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T10:54:58.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T10:54:58.124 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:54:58.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T10:54:58.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T10:54:58.225 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T10:54:58.225 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:54:58.225 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T10:54:58.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T10:54:58.323 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:54:58.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T10:54:58.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T10:54:58.423 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:54:58.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T10:54:58.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T10:54:58.525 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-16T10:54:58.611 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-16T10:54:58.611 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:54:58.612 INFO:tasks.cephadm:Cluster fsid is a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:54:58.612 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-16T10:54:58.612 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.b': '172.21.15.98', 'mon.c': '172.21.15.153'} 2024-03-16T10:54:58.612 INFO:tasks.cephadm:First mon is mon.a on smithi078 2024-03-16T10:54:58.612 INFO:tasks.cephadm:First mgr is x 2024-03-16T10:54:58.613 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-16T10:54:58.613 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2024-03-16T10:54:58.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-03-16T10:54:58.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-03-16T10:54:58.704 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-16T10:54:58.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:54:58.853 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-16T10:54:58.994 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:54:58.995 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-16T10:54:58.995 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T10:54:59.571 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 10:54 /home/ubuntu/cephtest/cephadm 2024-03-16T10:54:59.572 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T10:55:00.010 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 10:55 /home/ubuntu/cephtest/cephadm 2024-03-16T10:55:00.010 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T10:55:00.614 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 10:55 /home/ubuntu/cephtest/cephadm 2024-03-16T10:55:00.614 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T10:55:00.632 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T10:55:00.651 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T10:55:00.685 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-16T10:55:00.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T10:55:00.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T10:55:00.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T10:55:01.551 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:55:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:55:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:55:35.519 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T10:55:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T10:55:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T10:55:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2024-03-16T10:55:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T10:55:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T10:55:35.519 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T10:55:39.577 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T10:55:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T10:55:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T10:55:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-03-16T10:55:39.577 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T10:55:39.578 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T10:55:39.578 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T10:55:52.628 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T10:55:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T10:55:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T10:55:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-03-16T10:55:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T10:55:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-03-16T10:55:52.628 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T10:55:52.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2024-03-16T10:55:52.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-03-16T10:55:52.727 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-03-16T10:55:52.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2024-03-16T10:55:52.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-03-16T10:55:52.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-03-16T10:55:52.850 INFO:tasks.cephadm:Writing seed config... 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-16T10:55:52.851 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-16T10:55:52.852 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-16T10:55:52.853 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-16T10:55:52.854 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-16T10:55:52.854 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:55:52.854 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-16T10:55:52.872 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a1941e5e-e383-11ee-95c9-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-16T10:55:52.872 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a.service 2024-03-16T10:55:52.914 DEBUG:teuthology.orchestra.run.smithi078:mgr.x> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.x.service 2024-03-16T10:55:52.957 INFO:tasks.cephadm:Bootstrapping... 2024-03-16T10:55:52.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid a1941e5e-e383-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:55:53.152 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:55:53.152 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', 'a1941e5e-e383-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.78', '--skip-admin-label'] 2024-03-16T10:55:53.177 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T10:55:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-16T10:55:53.177 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying podman|docker is present... 2024-03-16T10:55:53.200 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T10:55:53.200 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying lvm2 is present... 2024-03-16T10:55:53.200 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying time synchronization is in place... 2024-03-16T10:55:53.207 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T10:55:53.208 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T10:55:53.213 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T10:55:53.213 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-16T10:55:53.219 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-03-16T10:55:53.225 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-03-16T10:55:53.225 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-03-16T10:55:53.225 INFO:teuthology.orchestra.run.smithi078.stdout:Repeating the final host check... 2024-03-16T10:55:53.249 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T10:55:53.249 INFO:teuthology.orchestra.run.smithi078.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-16T10:55:53.249 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl is present 2024-03-16T10:55:53.249 INFO:teuthology.orchestra.run.smithi078.stdout:lvcreate is present 2024-03-16T10:55:53.255 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T10:55:53.255 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T10:55:53.261 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T10:55:53.261 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-16T10:55:53.267 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-03-16T10:55:53.273 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-03-16T10:55:53.273 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-03-16T10:55:53.273 INFO:teuthology.orchestra.run.smithi078.stdout:Host looks OK 2024-03-16T10:55:53.273 INFO:teuthology.orchestra.run.smithi078.stdout:Cluster fsid: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:55:53.273 INFO:teuthology.orchestra.run.smithi078.stdout:Acquiring lock 140457352056256 on /run/cephadm/a1941e5e-e383-11ee-95c9-87774f69a715.lock 2024-03-16T10:55:53.274 INFO:teuthology.orchestra.run.smithi078.stdout:Lock 140457352056256 acquired on /run/cephadm/a1941e5e-e383-11ee-95c9-87774f69a715.lock 2024-03-16T10:55:53.274 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 3300 ... 2024-03-16T10:55:53.274 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 6789 ... 2024-03-16T10:55:53.274 INFO:teuthology.orchestra.run.smithi078.stdout:Base mon IP(s) is [172.21.15.78:3300, 172.21.15.78:6789], mon addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-03-16T10:55:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.78 metric 100 2024-03-16T10:55:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2024-03-16T10:55:53.280 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-16T10:55:53.280 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-16T10:55:53.280 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-03-16T10:55:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-16T10:55:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-16T10:55:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T10:55:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-16T10:55:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-16T10:55:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T10:55:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-03-16T10:55:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-03-16T10:55:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-16T10:55:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-16T10:55:53.284 INFO:teuthology.orchestra.run.smithi078.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:55:56.091 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T10:55:56.091 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:55:56.091 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Getting image source signatures 2024-03-16T10:55:56.091 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-16T10:55:56.091 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-16T10:55:56.091 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T10:55:56.091 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-16T10:55:57.070 INFO:teuthology.orchestra.run.smithi078.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T10:55:57.070 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T10:55:57.070 INFO:teuthology.orchestra.run.smithi078.stdout:Extracting ceph user uid/gid from container image... 2024-03-16T10:55:57.977 INFO:teuthology.orchestra.run.smithi078.stdout:stat: stdout 167 167 2024-03-16T10:55:57.977 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial keys... 2024-03-16T10:55:58.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQC+evVl1LgbEhAA9xaRlhBS7/mTj1ewv8L7Pw== 2024-03-16T10:55:59.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQC/evVlk/nTDBAAF//kNyKSUEL9WTprAHoHeQ== 2024-03-16T10:56:00.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQDAevVlqA0pChAAVBfFcGBQFfB93i6DV2FwEA== 2024-03-16T10:56:00.768 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial monmap... 2024-03-16T10:56:05.070 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: set fsid to a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T10:56:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:56:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mon... 2024-03-16T10:56:06.298 INFO:teuthology.orchestra.run.smithi078.stdout:create mon.a on 2024-03-16T10:56:06.507 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-16T10:56:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-16T10:56:06.823 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a1941e5e-e383-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a1941e5e-e383-11ee-95c9-87774f69a715.target. 2024-03-16T10:56:06.823 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a1941e5e-e383-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a1941e5e-e383-11ee-95c9-87774f69a715.target. 2024-03-16T10:56:07.065 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a 2024-03-16T10:56:07.065 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-16T10:56:07.235 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a1941e5e-e383-11ee-95c9-87774f69a715.target.wants/ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a1941e5e-e383-11ee-95c9-87774f69a715@.service. 2024-03-16T10:56:07.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:07 smithi078 podman[31836]: 2024-03-16 10:56:07.497136978 +0000 UTC m=+0.139387808 container create 5545487459d3a31dce1fc7faf5092a4bd6a9a99bba00d052ed57e18c8dc49b2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T10:56:07.771 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:56:07.772 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-16T10:56:07.779 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:56:07.779 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-16T10:56:07.779 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-16T10:56:07.779 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T10:56:07.779 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon to start... 2024-03-16T10:56:07.779 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon... 2024-03-16T10:56:07.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:07 smithi078 podman[31836]: 2024-03-16 10:56:07.691250866 +0000 UTC m=+0.333501710 container init 5545487459d3a31dce1fc7faf5092a4bd6a9a99bba00d052ed57e18c8dc49b2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-16T10:56:07.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:07 smithi078 podman[31836]: 2024-03-16 10:56:07.694259488 +0000 UTC m=+0.336510317 container start 5545487459d3a31dce1fc7faf5092a4bd6a9a99bba00d052ed57e18c8dc49b2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T10:56:07.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:07 smithi078 bash[31836]: 5545487459d3a31dce1fc7faf5092a4bd6a9a99bba00d052ed57e18c8dc49b2d 2024-03-16T10:56:07.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:07 smithi078 systemd[1]: Started Ceph mon.a for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:56:07.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:07 smithi078 ceph-mon[31856]: mkfs a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:07.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:07 smithi078 ceph-mon[31856]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: monmap epoch 1 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: last_changed 2024-03-16T10:56:01.082686+0000 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: min_mon_release 19 (squid) 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: election_strategy: 1 2024-03-16T10:56:08.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:56:08.866 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: fsmap 2024-03-16T10:56:08.866 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T10:56:08.866 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: mgrmap e1: no daemons active 2024-03-16T10:56:08.866 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:08 smithi078 ceph-mon[31856]: from='client.? 172.21.15.78:0/3266635153' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-16T10:56:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout cluster: 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout id: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout services: 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.51605s) 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout data: 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pgs: 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.198+0000 7f75c1995700 1 Processor -- start 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.198+0000 7f75c1995700 1 -- start start 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.199+0000 7f75c1995700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.199+0000 7f75c1995700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc1086f0 con 0x7f75bc107d00 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.199+0000 7f75baffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.199+0000 7f75baffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35842/0 (socket says 172.21.15.78:35842) 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.199+0000 7f75baffd700 1 -- 172.21.15.78:0/1980857727 learned_addr learned my addr 172.21.15.78:0/1980857727 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.199+0000 7f75baffd700 1 -- 172.21.15.78:0/1980857727 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75bc108f00 con 0x7f75bc107d00 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75baffd700 1 --2- 172.21.15.78:0/1980857727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc108120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75b00098a0 tx=0x7f75b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c22d610759db760 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75b9ffb700 1 -- 172.21.15.78:0/1980857727 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75b0004160 con 0x7f75bc107d00 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75b9ffb700 1 -- 172.21.15.78:0/1980857727 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b0004300 con 0x7f75bc107d00 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75b9ffb700 1 -- 172.21.15.78:0/1980857727 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75b00036a0 con 0x7f75bc107d00 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75c1995700 1 -- 172.21.15.78:0/1980857727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 msgr2=0x7f75bc108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75c1995700 1 --2- 172.21.15.78:0/1980857727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc108120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75b00098a0 tx=0x7f75b0009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75c1995700 1 -- 172.21.15.78:0/1980857727 shutdown_connections 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75c1995700 1 --2- 172.21.15.78:0/1980857727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc108120 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75c1995700 1 -- 172.21.15.78:0/1980857727 >> 172.21.15.78:0/1980857727 conn(0x7f75bc103340 msgr2=0x7f75bc105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75c1995700 1 -- 172.21.15.78:0/1980857727 shutdown_connections 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.200+0000 7f75c1995700 1 -- 172.21.15.78:0/1980857727 wait complete. 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.201+0000 7f75c1995700 1 Processor -- start 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.201+0000 7f75c1995700 1 -- start start 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.201+0000 7f75c1995700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.201+0000 7f75c1995700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc109430 con 0x7f75bc107d00 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.201+0000 7f75baffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75baffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc19b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35854/0 (socket says 172.21.15.78:35854) 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75baffd700 1 -- 172.21.15.78:0/3266635153 learned_addr learned my addr 172.21.15.78:0/3266635153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75baffd700 1 -- 172.21.15.78:0/3266635153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75bc19b7e0 con 0x7f75bc107d00 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75baffd700 1 --2- 172.21.15.78:0/3266635153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc19b2a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f75bc108990 tx=0x7f75b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:08.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75c0993700 1 -- 172.21.15.78:0/3266635153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75b0003db0 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75c0993700 1 -- 172.21.15.78:0/3266635153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b0011630 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75bc19ba70 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75c0993700 1 -- 172.21.15.78:0/3266635153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75b0011930 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75bc19e760 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75c0993700 1 -- 172.21.15.78:0/3266635153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75b0011ad0 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.202+0000 7f75c0993700 1 -- 172.21.15.78:0/3266635153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f75b001b450 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.203+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75bc063900 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.205+0000 7f75c0993700 1 -- 172.21.15.78:0/3266635153 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f75b0004260 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.276+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f75bc02dcc0 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.276+0000 7f75c0993700 1 -- 172.21.15.78:0/3266635153 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f75b0025480 con 0x7f75bc107d00 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.277+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 msgr2=0x7f75bc19b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.277+0000 7f75c1995700 1 --2- 172.21.15.78:0/3266635153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc19b2a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f75bc108990 tx=0x7f75b0004750 comp rx=0 tx=0).stop 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.277+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 shutdown_connections 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.277+0000 7f75c1995700 1 --2- 172.21.15.78:0/3266635153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75bc107d00 0x7f75bc19b2a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:08.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.277+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 >> 172.21.15.78:0/3266635153 conn(0x7f75bc103340 msgr2=0x7f75bc191f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:08.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.277+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 shutdown_connections 2024-03-16T10:56:08.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:08.277+0000 7f75c1995700 1 -- 172.21.15.78:0/3266635153 wait complete. 2024-03-16T10:56:08.877 INFO:teuthology.orchestra.run.smithi078.stdout:mon is available 2024-03-16T10:56:08.877 INFO:teuthology.orchestra.run.smithi078.stdout:Assimilating anything we can from ceph.conf... 2024-03-16T10:56:09.987 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:09 smithi078 ceph-mon[31856]: from='client.? 172.21.15.78:0/2262545665' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T10:56:09.987 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:09 smithi078 ceph-mon[31856]: from='client.? 172.21.15.78:0/2262545665' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T10:56:09.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:09.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T10:56:09.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:09.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-03-16T10:56:09.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T10:56:09.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T10:56:09.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.322+0000 7f5b1080d700 1 Processor -- start 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.322+0000 7f5b1080d700 1 -- start start 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b1080d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c10abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b1080d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c063590 con 0x7f5b0c1087c0 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c10abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c10abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35856/0 (socket says 172.21.15.78:35856) 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0a59c700 1 -- 172.21.15.78:0/1296209927 learned_addr learned my addr 172.21.15.78:0/1296209927 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0a59c700 1 -- 172.21.15.78:0/1296209927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0c10b0f0 con 0x7f5b0c1087c0 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0a59c700 1 --2- 172.21.15.78:0/1296209927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c10abb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0098a0 tx=0x7f5afc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8c837665e70923 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0959a700 1 -- 172.21.15.78:0/1296209927 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5afc004160 con 0x7f5b0c1087c0 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0959a700 1 -- 172.21.15.78:0/1296209927 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5afc004300 con 0x7f5b0c1087c0 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.323+0000 7f5b0959a700 1 -- 172.21.15.78:0/1296209927 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5afc0036a0 con 0x7f5b0c1087c0 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.324+0000 7f5b1080d700 1 -- 172.21.15.78:0/1296209927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 msgr2=0x7f5b0c10abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.324+0000 7f5b1080d700 1 --2- 172.21.15.78:0/1296209927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c10abb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0098a0 tx=0x7f5afc009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.324+0000 7f5b1080d700 1 -- 172.21.15.78:0/1296209927 shutdown_connections 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.324+0000 7f5b1080d700 1 --2- 172.21.15.78:0/1296209927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c10abb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.324+0000 7f5b1080d700 1 -- 172.21.15.78:0/1296209927 >> 172.21.15.78:0/1296209927 conn(0x7f5b0c1021b0 msgr2=0x7f5b0c1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.324+0000 7f5b1080d700 1 -- 172.21.15.78:0/1296209927 shutdown_connections 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.324+0000 7f5b1080d700 1 -- 172.21.15.78:0/1296209927 wait complete. 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b1080d700 1 Processor -- start 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b1080d700 1 -- start start 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b1080d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c19b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b1080d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10b560 con 0x7f5b0c1087c0 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b0a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c19b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b0a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c19b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35862/0 (socket says 172.21.15.78:35862) 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b0a59c700 1 -- 172.21.15.78:0/2262545665 learned_addr learned my addr 172.21.15.78:0/2262545665 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b0a59c700 1 -- 172.21.15.78:0/2262545665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0c19b7c0 con 0x7f5b0c1087c0 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.325+0000 7f5b0a59c700 1 --2- 172.21.15.78:0/2262545665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c19b280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5afc000c00 tx=0x7f5afc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.326+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5afc003db0 con 0x7f5b0c1087c0 2024-03-16T10:56:09.999 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.326+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5afc011630 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.326+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0c19ba50 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.326+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5afc011950 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.326+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0c19e6f0 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.326+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5afc011af0 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.326+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5afc01c450 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.327+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b0c063900 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.329+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5afc032020 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.396+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f5b0c02dd80 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.399+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+456 (secure 0 0 0) 0x7f5afc004260 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.400+0000 7f5afb7fe700 1 -- 172.21.15.78:0/2262545665 <== mon.0 v2:172.21.15.78:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5afc01c6a0 con 0x7f5b0c1087c0 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.401+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 msgr2=0x7f5b0c19b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.401+0000 7f5b1080d700 1 --2- 172.21.15.78:0/2262545665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c19b280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5afc000c00 tx=0x7f5afc004750 comp rx=0 tx=0).stop 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.401+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 shutdown_connections 2024-03-16T10:56:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.401+0000 7f5b1080d700 1 --2- 172.21.15.78:0/2262545665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b0c1087c0 0x7f5b0c19b280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.401+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 >> 172.21.15.78:0/2262545665 conn(0x7f5b0c1021b0 msgr2=0x7f5b0c104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.401+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 shutdown_connections 2024-03-16T10:56:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:09.401+0000 7f5b1080d700 1 -- 172.21.15.78:0/2262545665 wait complete. 2024-03-16T10:56:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:Generating new minimal ceph.conf... 2024-03-16T10:56:11.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:10 smithi078 ceph-mon[31856]: from='client.? 172.21.15.78:0/1645686322' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:56:11.567 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.411+0000 7f8f34a22700 1 Processor -- start 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.411+0000 7f8f34a22700 1 -- start start 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.411+0000 7f8f34a22700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3007c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.411+0000 7f8f34a22700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3007c830 con 0x7f8f3007de00 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.411+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3007c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.411+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3007c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35868/0 (socket says 172.21.15.78:35868) 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.411+0000 7f8f2e59c700 1 -- 172.21.15.78:0/2168479643 learned_addr learned my addr 172.21.15.78:0/2168479643 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.412+0000 7f8f2e59c700 1 -- 172.21.15.78:0/2168479643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3007c9b0 con 0x7f8f3007de00 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.412+0000 7f8f2e59c700 1 --2- 172.21.15.78:0/2168479643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3007c260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009e80 tx=0x7f8f2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385f75cd24e0f307 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.412+0000 7f8f2d59a700 1 -- 172.21.15.78:0/2168479643 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f28017070 con 0x7f8f3007de00 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.412+0000 7f8f2d59a700 1 -- 172.21.15.78:0/2168479643 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8f2800b8c0 con 0x7f8f3007de00 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.412+0000 7f8f2d59a700 1 -- 172.21.15.78:0/2168479643 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f28004520 con 0x7f8f3007de00 2024-03-16T10:56:11.568 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 -- 172.21.15.78:0/2168479643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 msgr2=0x7f8f3007c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 --2- 172.21.15.78:0/2168479643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3007c260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009e80 tx=0x7f8f2800b330 comp rx=0 tx=0).stop 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 -- 172.21.15.78:0/2168479643 shutdown_connections 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 --2- 172.21.15.78:0/2168479643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3007c260 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 -- 172.21.15.78:0/2168479643 >> 172.21.15.78:0/2168479643 conn(0x7f8f30103340 msgr2=0x7f8f30105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 -- 172.21.15.78:0/2168479643 shutdown_connections 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 -- 172.21.15.78:0/2168479643 wait complete. 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.413+0000 7f8f34a22700 1 Processor -- start 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f34a22700 1 -- start start 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f34a22700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f34a22700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3010e200 con 0x7f8f3007de00 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3019dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35880/0 (socket says 172.21.15.78:35880) 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f2e59c700 1 -- 172.21.15.78:0/1645686322 learned_addr learned my addr 172.21.15.78:0/1645686322 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f2e59c700 1 -- 172.21.15.78:0/1645686322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3019e1f0 con 0x7f8f3007de00 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f2e59c700 1 --2- 172.21.15.78:0/1645686322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3019dcb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8f280042b0 tx=0x7f8f28004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f1f7fe700 1 -- 172.21.15.78:0/1645686322 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f28020020 con 0x7f8f3007de00 2024-03-16T10:56:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f1f7fe700 1 -- 172.21.15.78:0/1645686322 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8f2800bba0 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f3019e480 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f1f7fe700 1 -- 172.21.15.78:0/1645686322 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f28017040 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.414+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f301a1170 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.415+0000 7f8f1f7fe700 1 -- 172.21.15.78:0/1645686322 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8f28023020 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.415+0000 7f8f1f7fe700 1 -- 172.21.15.78:0/1645686322 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8f2800f050 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.416+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f30063900 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.418+0000 7f8f1f7fe700 1 -- 172.21.15.78:0/1645686322 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8f2802a020 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.485+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8f301a1730 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.485+0000 7f8f1f7fe700 1 -- 172.21.15.78:0/1645686322 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f8f2800b860 con 0x7f8f3007de00 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.486+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 msgr2=0x7f8f3019dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.486+0000 7f8f34a22700 1 --2- 172.21.15.78:0/1645686322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3019dcb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8f280042b0 tx=0x7f8f28004670 comp rx=0 tx=0).stop 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.486+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 shutdown_connections 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.486+0000 7f8f34a22700 1 --2- 172.21.15.78:0/1645686322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f3007de00 0x7f8f3019dcb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.486+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 >> 172.21.15.78:0/1645686322 conn(0x7f8f30103340 msgr2=0x7f8f30103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:11.571 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.486+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 shutdown_connections 2024-03-16T10:56:11.571 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:10.486+0000 7f8f34a22700 1 -- 172.21.15.78:0/1645686322 wait complete. 2024-03-16T10:56:11.571 INFO:teuthology.orchestra.run.smithi078.stdout:Restarting the monitor... 2024-03-16T10:56:11.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:11 smithi078 systemd[1]: Stopping Ceph mon.a for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T10:56:12.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:11 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a[31852]: 2024-03-16T10:56:11.837+0000 7f09df691700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:56:12.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:11 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a[31852]: 2024-03-16T10:56:11.837+0000 7f09df691700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T10:56:12.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:12 smithi078 podman[32101]: 2024-03-16 10:56:12.093673313 +0000 UTC m=+0.366390923 container died 5545487459d3a31dce1fc7faf5092a4bd6a9a99bba00d052ed57e18c8dc49b2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T10:56:12.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:12 smithi078 podman[32101]: 2024-03-16 10:56:12.471081218 +0000 UTC m=+0.743798827 container cleanup 5545487459d3a31dce1fc7faf5092a4bd6a9a99bba00d052ed57e18c8dc49b2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:56:12.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:12 smithi078 bash[32101]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a 2024-03-16T10:56:13.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:12 smithi078 podman[32116]: 2024-03-16 10:56:12.747032499 +0000 UTC m=+0.657787397 container remove 5545487459d3a31dce1fc7faf5092a4bd6a9a99bba00d052ed57e18c8dc49b2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:56:13.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:12 smithi078 systemd[1]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-16T10:56:13.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:12 smithi078 systemd[1]: Stopped Ceph mon.a for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:56:13.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:12 smithi078 systemd[1]: Starting Ceph mon.a for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T10:56:13.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 podman[32216]: 2024-03-16 10:56:12.985825229 +0000 UTC m=+0.018508229 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:56:13.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 podman[32216]: 2024-03-16 10:56:13.089612935 +0000 UTC m=+0.122295921 container create b5c01b21f69c01bece854b37128e32a817931f264d2a627a234ae336acb0e572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T10:56:13.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 podman[32216]: 2024-03-16 10:56:13.342171005 +0000 UTC m=+0.374854001 container init b5c01b21f69c01bece854b37128e32a817931f264d2a627a234ae336acb0e572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:56:13.421 INFO:teuthology.orchestra.run.smithi078.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-16T10:56:13.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 podman[32216]: 2024-03-16 10:56:13.344976138 +0000 UTC m=+0.377659124 container start b5c01b21f69c01bece854b37128e32a817931f264d2a627a234ae336acb0e572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: pidfile_write: ignore empty --pid-file 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: load: jerasure load: lrc 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: RocksDB version: 7.9.2 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Git sha 0 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: DB SUMMARY 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: DB Session ID: 1K29J6EYCS1J97LE5TRB 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: CURRENT file: CURRENT 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-16T10:56:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104146 ; 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.error_if_exists: 0 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.create_if_missing: 0 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.paranoid_checks: 1 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.env: 0x5611a9b2fc60 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.info_log: 0x5611ab413ac0 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.statistics: (nil) 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.use_fsync: 0 2024-03-16T10:56:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_log_file_size: 0 2024-03-16T10:56:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T10:56:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T10:56:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T10:56:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T10:56:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.allow_fallocate: 1 2024-03-16T10:56:13.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T10:56:13.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T10:56:13.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.use_direct_reads: 0 2024-03-16T10:56:13.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T10:56:13.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T10:56:13.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.db_log_dir: 2024-03-16T10:56:13.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.wal_dir: 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.write_buffer_manager: 0x5611ab3e61e0 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T10:56:13.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.unordered_write: 0 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.row_cache: None 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.wal_filter: None 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.two_write_queues: 0 2024-03-16T10:56:13.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.wal_compression: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.atomic_flush: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.log_readahead_size: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T10:56:13.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_background_jobs: 2 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_background_compactions: -1 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_subcompactions: 1 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_open_files: -1 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T10:56:13.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_background_flushes: -1 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Compression algorithms supported: 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kZSTD supported: 0 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kXpressCompression supported: 0 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kZlibCompression supported: 1 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kSnappyCompression supported: 1 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kLZ4Compression supported: 1 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: kBZip2Compression supported: 0 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T10:56:13.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.merge_operator: 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_filter: None 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5611ab413bc0) 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2024-03-16T10:56:13.637 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x5611ab419090 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2024-03-16T10:56:13.638 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T10:56:13.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression: NoCompression 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.num_levels: 7 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T10:56:13.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T10:56:13.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T10:56:13.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.table_properties_collectors: 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.inplace_update_support: 0 2024-03-16T10:56:13.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.bloom_locality: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.max_successive_merges: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.ttl: 2592000 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.enable_blob_files: false 2024-03-16T10:56:13.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.min_blob_size: 0 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a454aff1-227d-4c9b-bd77-a4d86eddebbd 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586573373835, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586573374819, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99326, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97298, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13943, "raw_average_key_size": 49, "raw_value_size": 89688, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710586573, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a454aff1-227d-4c9b-bd77-a4d86eddebbd", "db_session_id": "1K29J6EYCS1J97LE5TRB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-16T10:56:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586573374878, "job": 1, "event": "recovery_finished"} 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5611ab50c000 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: DB pointer 0x5611ab4f6000 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 2/0 98.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 98.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-16T10:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 32.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 32.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x5611ab419090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???) e1 preinit fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:13.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???).mds e1 new map 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???).mds e1 print_map 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 bash[32216]: b5c01b21f69c01bece854b37128e32a817931f264d2a627a234ae336acb0e572 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 systemd[1]: Started Ceph mon.a for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: monmap epoch 1 2024-03-16T10:56:13.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: last_changed 2024-03-16T10:56:01.082686+0000 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: min_mon_release 19 (squid) 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: election_strategy: 1 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: fsmap 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T10:56:13.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:13 smithi078 ceph-mon[32235]: mgrmap e1: no daemons active 2024-03-16T10:56:14.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.837+0000 7fe9d2a88700 1 Processor -- start 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.837+0000 7fe9d2a88700 1 -- start start 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.837+0000 7fe9d2a88700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc10ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.837+0000 7fe9d2a88700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc07bdf0 con 0x7fe9cc108790 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.837+0000 7fe9d0824700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc10ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.837+0000 7fe9d0824700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc10ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35892/0 (socket says 172.21.15.78:35892) 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.837+0000 7fe9d0824700 1 -- 172.21.15.78:0/3912443347 learned_addr learned my addr 172.21.15.78:0/3912443347 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9d0824700 1 -- 172.21.15.78:0/3912443347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc10b0c0 con 0x7fe9cc108790 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9d0824700 1 --2- 172.21.15.78:0/3912443347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc10ab80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00098a0 tx=0x7fe9c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5ecadf5339d498 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9cb7fe700 1 -- 172.21.15.78:0/3912443347 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9c0004160 con 0x7fe9cc108790 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9cb7fe700 1 -- 172.21.15.78:0/3912443347 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe9c0004300 con 0x7fe9cc108790 2024-03-16T10:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9cb7fe700 1 -- 172.21.15.78:0/3912443347 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9c0011a80 con 0x7fe9cc108790 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9d2a88700 1 -- 172.21.15.78:0/3912443347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 msgr2=0x7fe9cc10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9d2a88700 1 --2- 172.21.15.78:0/3912443347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc10ab80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00098a0 tx=0x7fe9c0009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.838+0000 7fe9d2a88700 1 -- 172.21.15.78:0/3912443347 shutdown_connections 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 --2- 172.21.15.78:0/3912443347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc10ab80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 -- 172.21.15.78:0/3912443347 >> 172.21.15.78:0/3912443347 conn(0x7fe9cc102220 msgr2=0x7fe9cc104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 -- 172.21.15.78:0/3912443347 shutdown_connections 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 -- 172.21.15.78:0/3912443347 wait complete. 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 Processor -- start 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 -- start start 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d2a88700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc10bfa0 con 0x7fe9cc108790 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d0824700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d0824700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35896/0 (socket says 172.21.15.78:35896) 2024-03-16T10:56:14.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.839+0000 7fe9d0824700 1 -- 172.21.15.78:0/1053366357 learned_addr learned my addr 172.21.15.78:0/1053366357 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9d0824700 1 -- 172.21.15.78:0/1053366357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc19c000 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9d0824700 1 --2- 172.21.15.78:0/1053366357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc19bac0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0006af0 tx=0x7fe9c0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9c9ffb700 1 -- 172.21.15.78:0/1053366357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9c00039e0 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9c9ffb700 1 -- 172.21.15.78:0/1053366357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe9c0003b80 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc19c290 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9c9ffb700 1 -- 172.21.15.78:0/1053366357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9c00196a0 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc19ef80 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9c9ffb700 1 -- 172.21.15.78:0/1053366357 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe9c002d440 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.840+0000 7fe9c9ffb700 1 -- 172.21.15.78:0/1053366357 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe9c00247c0 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.841+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9cc063900 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.843+0000 7fe9c9ffb700 1 -- 172.21.15.78:0/1053366357 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe9c0043dd0 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.909+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe9cc19f150 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.911+0000 7fe9c9ffb700 1 -- 172.21.15.78:0/1053366357 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe9c0032020 con 0x7fe9cc108790 2024-03-16T10:56:14.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.912+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 msgr2=0x7fe9cc19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:14.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.912+0000 7fe9d2a88700 1 --2- 172.21.15.78:0/1053366357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc19bac0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0006af0 tx=0x7fe9c0011c40 comp rx=0 tx=0).stop 2024-03-16T10:56:14.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.912+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 shutdown_connections 2024-03-16T10:56:14.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.912+0000 7fe9d2a88700 1 --2- 172.21.15.78:0/1053366357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9cc108790 0x7fe9cc19bac0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:14.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.912+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 >> 172.21.15.78:0/1053366357 conn(0x7fe9cc102220 msgr2=0x7fe9cc102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:14.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.912+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 shutdown_connections 2024-03-16T10:56:14.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:13.912+0000 7fe9d2a88700 1 -- 172.21.15.78:0/1053366357 wait complete. 2024-03-16T10:56:14.442 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-16T10:56:14.456 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:56:14.456 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mgr... 2024-03-16T10:56:14.456 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-16T10:56:14.457 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-16T10:56:14.707 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.x 2024-03-16T10:56:14.708 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-16T10:56:14.861 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a1941e5e-e383-11ee-95c9-87774f69a715.target.wants/ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-a1941e5e-e383-11ee-95c9-87774f69a715@.service. 2024-03-16T10:56:15.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:14 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1053366357' entity='client.admin' 2024-03-16T10:56:15.323 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:56:15.323 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-16T10:56:15.330 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:56:15.330 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-16T10:56:15.330 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-16T10:56:15.330 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T10:56:15.338 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:56:15.338 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-16T10:56:15.345 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:56:15.345 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-16T10:56:15.346 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-16T10:56:15.346 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-16T10:56:15.346 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr to start... 2024-03-16T10:56:15.346 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr... 2024-03-16T10:56:15.461 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:15 smithi078 systemd[1]: Started Ceph mgr.x for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:56:16.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:15 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2300855493' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:56:16.060 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:15 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:15.828+0000 7fd73d0fa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:56:16.060 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:15 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:15.926+0000 7fd73d0fa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:56:16.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:16.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:56:16.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T10:56:16.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:56:16.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:56:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:56:16.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:56:16.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:56:16.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:56:16.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:56:16.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:16.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:56:16.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:56:16.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:56:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:56:07.774508+0000", 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.771+0000 7f5784be1700 1 Processor -- start 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.771+0000 7f5784be1700 1 -- start start 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.771+0000 7f5784be1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578010ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.771+0000 7f5784be1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f578007bdf0 con 0x7f5780108780 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578010ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578010ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53106/0 (socket says 172.21.15.78:53106) 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577e59c700 1 -- 172.21.15.78:0/1135779251 learned_addr learned my addr 172.21.15.78:0/1135779251 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577e59c700 1 -- 172.21.15.78:0/1135779251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578010b0b0 con 0x7f5780108780 2024-03-16T10:56:16.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577e59c700 1 --2- 172.21.15.78:0/1135779251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578010ab70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57700098a0 tx=0x7f5770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0eeff8031d7c1cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577d59a700 1 -- 172.21.15.78:0/1135779251 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5770004160 con 0x7f5780108780 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577d59a700 1 -- 172.21.15.78:0/1135779251 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5770004300 con 0x7f5780108780 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.772+0000 7f577d59a700 1 -- 172.21.15.78:0/1135779251 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5770011a80 con 0x7f5780108780 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.773+0000 7f5784be1700 1 -- 172.21.15.78:0/1135779251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 msgr2=0x7f578010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.773+0000 7f5784be1700 1 --2- 172.21.15.78:0/1135779251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578010ab70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57700098a0 tx=0x7f5770009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.773+0000 7f5784be1700 1 -- 172.21.15.78:0/1135779251 shutdown_connections 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.773+0000 7f5784be1700 1 --2- 172.21.15.78:0/1135779251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578010ab70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.773+0000 7f5784be1700 1 -- 172.21.15.78:0/1135779251 >> 172.21.15.78:0/1135779251 conn(0x7f57801021d0 msgr2=0x7f57801045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.773+0000 7f5784be1700 1 -- 172.21.15.78:0/1135779251 shutdown_connections 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.773+0000 7f5784be1700 1 -- 172.21.15.78:0/1135779251 wait complete. 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.774+0000 7f5784be1700 1 Processor -- start 2024-03-16T10:56:16.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.774+0000 7f5784be1700 1 -- start start 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.774+0000 7f5784be1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.774+0000 7f5784be1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f578010bf90 con 0x7f5780108780 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.774+0000 7f577e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.774+0000 7f577e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578019baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53110/0 (socket says 172.21.15.78:53110) 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.774+0000 7f577e59c700 1 -- 172.21.15.78:0/2300855493 learned_addr learned my addr 172.21.15.78:0/2300855493 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f577e59c700 1 -- 172.21.15.78:0/2300855493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578019bfe0 con 0x7f5780108780 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f577e59c700 1 --2- 172.21.15.78:0/2300855493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578019baa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5770006af0 tx=0x7f5770011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f576f7fe700 1 -- 172.21.15.78:0/2300855493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57700039e0 con 0x7f5780108780 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f576f7fe700 1 -- 172.21.15.78:0/2300855493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5770003b80 con 0x7f5780108780 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f578019c270 con 0x7f5780108780 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f576f7fe700 1 -- 172.21.15.78:0/2300855493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57700196a0 con 0x7f5780108780 2024-03-16T10:56:16.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f578019ef60 con 0x7f5780108780 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.775+0000 7f576f7fe700 1 -- 172.21.15.78:0/2300855493 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f577002d440 con 0x7f5780108780 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.776+0000 7f576f7fe700 1 -- 172.21.15.78:0/2300855493 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f57700244c0 con 0x7f5780108780 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.776+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5780063900 con 0x7f5780108780 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.780+0000 7f576f7fe700 1 -- 172.21.15.78:0/2300855493 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5770043dd0 con 0x7f5780108780 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.858+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f578019f130 con 0x7f5780108780 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.859+0000 7f576f7fe700 1 -- 172.21.15.78:0/2300855493 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5770032020 con 0x7f5780108780 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.860+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 msgr2=0x7f578019baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.860+0000 7f5784be1700 1 --2- 172.21.15.78:0/2300855493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578019baa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5770006af0 tx=0x7f5770011c40 comp rx=0 tx=0).stop 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.860+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 shutdown_connections 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.860+0000 7f5784be1700 1 --2- 172.21.15.78:0/2300855493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5780108780 0x7f578019baa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.860+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 >> 172.21.15.78:0/2300855493 conn(0x7f57801021d0 msgr2=0x7f5780102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.860+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 shutdown_connections 2024-03-16T10:56:16.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:15.860+0000 7f5784be1700 1 -- 172.21.15.78:0/2300855493 wait complete. 2024-03-16T10:56:16.406 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (1/15)... 2024-03-16T10:56:16.660 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:16.568+0000 7fd73d0fa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:56:16.660 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:16.659+0000 7fd73d0fa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:56:16.970 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:16.797+0000 7fd73d0fa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:56:16.971 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:16.889+0000 7fd73d0fa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:56:16.971 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:16.969+0000 7fd73d0fa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:56:17.523 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:17.238+0000 7fd73d0fa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:56:17.523 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:17.345+0000 7fd73d0fa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:56:17.523 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:17.429+0000 7fd73d0fa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:56:17.523 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:17.521+0000 7fd73d0fa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:56:17.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:17.694+0000 7fd73d0fa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:56:18.777 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:18 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:18.658+0000 7fd73d0fa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:56:19.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:19 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3556511512' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:56:19.059 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:18 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:18.790+0000 7fd73d0fa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:56:19.420 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:19.135+0000 7fd73d0fa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:56:19.420 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:19.224+0000 7fd73d0fa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:56:19.420 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:19.419+0000 7fd73d0fa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:56:19.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:56:19.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:56:19.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:56:19.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:56:19.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:56:19.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:56:07.774508+0000", 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:19.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.876+0000 7fea49449700 1 Processor -- start 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.876+0000 7fea49449700 1 -- start start 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.876+0000 7fea49449700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44107010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.876+0000 7fea49449700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea441075e0 con 0x7fea44106bf0 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.876+0000 7fea42ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.876+0000 7fea42ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53122/0 (socket says 172.21.15.78:53122) 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.876+0000 7fea42ffd700 1 -- 172.21.15.78:0/1441897433 learned_addr learned my addr 172.21.15.78:0/1441897433 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.877+0000 7fea42ffd700 1 -- 172.21.15.78:0/1441897433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea44107df0 con 0x7fea44106bf0 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.877+0000 7fea42ffd700 1 --2- 172.21.15.78:0/1441897433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44107010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea340098a0 tx=0x7fea34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452721cd839d3b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.877+0000 7fea41ffb700 1 -- 172.21.15.78:0/1441897433 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea34004160 con 0x7fea44106bf0 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.877+0000 7fea41ffb700 1 -- 172.21.15.78:0/1441897433 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fea34004300 con 0x7fea44106bf0 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.877+0000 7fea41ffb700 1 -- 172.21.15.78:0/1441897433 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea34011a80 con 0x7fea44106bf0 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 -- 172.21.15.78:0/1441897433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 msgr2=0x7fea44107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 --2- 172.21.15.78:0/1441897433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44107010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea340098a0 tx=0x7fea34009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:19.631 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 -- 172.21.15.78:0/1441897433 shutdown_connections 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 --2- 172.21.15.78:0/1441897433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44107010 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 -- 172.21.15.78:0/1441897433 >> 172.21.15.78:0/1441897433 conn(0x7fea441021b0 msgr2=0x7fea441045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 -- 172.21.15.78:0/1441897433 shutdown_connections 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 -- 172.21.15.78:0/1441897433 wait complete. 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.878+0000 7fea49449700 1 Processor -- start 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea49449700 1 -- start start 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea49449700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44063190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea49449700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea4410b020 con 0x7fea44106bf0 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea42ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44063190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea42ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44063190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53138/0 (socket says 172.21.15.78:53138) 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea42ffd700 1 -- 172.21.15.78:0/3556511512 learned_addr learned my addr 172.21.15.78:0/3556511512 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea42ffd700 1 -- 172.21.15.78:0/3556511512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4419e8c0 con 0x7fea44106bf0 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea42ffd700 1 --2- 172.21.15.78:0/3556511512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44063190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fea34000c00 tx=0x7fea34004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.879+0000 7fea2bfff700 1 -- 172.21.15.78:0/3556511512 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea340039c0 con 0x7fea44106bf0 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.880+0000 7fea2bfff700 1 -- 172.21.15.78:0/3556511512 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fea34003b60 con 0x7fea44106bf0 2024-03-16T10:56:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.880+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4419ea90 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.880+0000 7fea2bfff700 1 -- 172.21.15.78:0/3556511512 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea3401a6f0 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.880+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4419ee30 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.880+0000 7fea2bfff700 1 -- 172.21.15.78:0/3556511512 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea3401ac20 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.880+0000 7fea2bfff700 1 -- 172.21.15.78:0/3556511512 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fea34019700 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.881+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea44063900 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.884+0000 7fea2bfff700 1 -- 172.21.15.78:0/3556511512 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fea34004300 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.956+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fea4419f5e0 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.956+0000 7fea2bfff700 1 -- 172.21.15.78:0/3556511512 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fea34024a40 con 0x7fea44106bf0 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.957+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 msgr2=0x7fea44063190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.957+0000 7fea49449700 1 --2- 172.21.15.78:0/3556511512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44063190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fea34000c00 tx=0x7fea34004290 comp rx=0 tx=0).stop 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.957+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 shutdown_connections 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.957+0000 7fea49449700 1 --2- 172.21.15.78:0/3556511512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea44106bf0 0x7fea44063190 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.957+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 >> 172.21.15.78:0/3556511512 conn(0x7fea441021b0 msgr2=0x7fea44102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.958+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 shutdown_connections 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:18.958+0000 7fea49449700 1 -- 172.21.15.78:0/3556511512 wait complete. 2024-03-16T10:56:19.633 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (2/15)... 2024-03-16T10:56:19.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:19.607+0000 7fd73d0fa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:56:20.173 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:19.893+0000 7fd73d0fa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:56:20.173 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:20 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:20.172+0000 7fd73d0fa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:56:20.559 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:20 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:20.407+0000 7fd73d0fa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:56:21.559 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:21 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:21.198+0000 7fd73d0fa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:56:22.008 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:21 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:21.762+0000 7fd73d0fa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:56:22.008 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:21 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:21.860+0000 7fd73d0fa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:56:22.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:22.082+0000 7fd73d0fa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: Activating manager daemon x 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: mgrmap e2: x(active, starting, since 0.0032008s) 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: Manager daemon x is now available 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3571622535' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:56:22.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' 2024-03-16T10:56:22.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:22 smithi078 ceph-mon[32235]: from='mgr.14100 172.21.15.78:0/1031784788' entity='mgr.x' 2024-03-16T10:56:22.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:22.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:56:22.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T10:56:22.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:56:22.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:56:22.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:56:22.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:22.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:56:22.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:56:22.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:56:22.751 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:56:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:56:07.774508+0000", 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.047+0000 7efd85cd4700 1 Processor -- start 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.047+0000 7efd85cd4700 1 -- start start 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.047+0000 7efd85cd4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8007c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.047+0000 7efd85cd4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efd8007c870 con 0x7efd8007de40 2024-03-16T10:56:22.753 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8007c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8007c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53140/0 (socket says 172.21.15.78:53140) 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7f7fe700 1 -- 172.21.15.78:0/1655897362 learned_addr learned my addr 172.21.15.78:0/1655897362 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7f7fe700 1 -- 172.21.15.78:0/1655897362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8007c9f0 con 0x7efd8007de40 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7f7fe700 1 --2- 172.21.15.78:0/1655897362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8007c2a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efd740098a0 tx=0x7efd74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1fb3d0863077e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7e7fc700 1 -- 172.21.15.78:0/1655897362 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd74004160 con 0x7efd8007de40 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7e7fc700 1 -- 172.21.15.78:0/1655897362 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efd74004300 con 0x7efd8007de40 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.048+0000 7efd7e7fc700 1 -- 172.21.15.78:0/1655897362 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd74011a80 con 0x7efd8007de40 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.049+0000 7efd85cd4700 1 -- 172.21.15.78:0/1655897362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 msgr2=0x7efd8007c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.049+0000 7efd85cd4700 1 --2- 172.21.15.78:0/1655897362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8007c2a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efd740098a0 tx=0x7efd74009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.049+0000 7efd85cd4700 1 -- 172.21.15.78:0/1655897362 shutdown_connections 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.049+0000 7efd85cd4700 1 --2- 172.21.15.78:0/1655897362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8007c2a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.049+0000 7efd85cd4700 1 -- 172.21.15.78:0/1655897362 >> 172.21.15.78:0/1655897362 conn(0x7efd80103270 msgr2=0x7efd80105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.049+0000 7efd85cd4700 1 -- 172.21.15.78:0/1655897362 shutdown_connections 2024-03-16T10:56:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.049+0000 7efd85cd4700 1 -- 172.21.15.78:0/1655897362 wait complete. 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.050+0000 7efd85cd4700 1 Processor -- start 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.050+0000 7efd85cd4700 1 -- start start 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.050+0000 7efd85cd4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.050+0000 7efd85cd4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010e170 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.050+0000 7efd7f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.050+0000 7efd7f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8019dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53142/0 (socket says 172.21.15.78:53142) 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.050+0000 7efd7f7fe700 1 -- 172.21.15.78:0/3571622535 learned_addr learned my addr 172.21.15.78:0/3571622535 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd7f7fe700 1 -- 172.21.15.78:0/3571622535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8019e260 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd7f7fe700 1 --2- 172.21.15.78:0/3571622535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8019dd20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efd74006af0 tx=0x7efd74011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd74003930 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efd74003ad0 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8019e4f0 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd740195e0 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd801a11d0 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efd74019c40 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.051+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efd74024b10 con 0x7efd8007de40 2024-03-16T10:56:22.755 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.052+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd80063900 con 0x7efd8007de40 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.056+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7efd74043dd0 con 0x7efd8007de40 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.098+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 2) ==== 45862+0+0 (secure 0 0 0) 0x7efd7401a430 con 0x7efd8007de40 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.131+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efd801a13a0 con 0x7efd8007de40 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.134+0000 7efd7cff9700 1 -- 172.21.15.78:0/3571622535 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efd74022340 con 0x7efd8007de40 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.135+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 msgr2=0x7efd8019dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.135+0000 7efd85cd4700 1 --2- 172.21.15.78:0/3571622535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8019dd20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efd74006af0 tx=0x7efd74011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.135+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 shutdown_connections 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.135+0000 7efd85cd4700 1 --2- 172.21.15.78:0/3571622535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd8007de40 0x7efd8019dd20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.135+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 >> 172.21.15.78:0/3571622535 conn(0x7efd80103270 msgr2=0x7efd80103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.135+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 shutdown_connections 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:22.135+0000 7efd85cd4700 1 -- 172.21.15.78:0/3571622535 wait complete. 2024-03-16T10:56:22.756 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (3/15)... 2024-03-16T10:56:24.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:24 smithi078 ceph-mon[32235]: mgrmap e3: x(active, since 1.00745s) 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:25.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:56:25.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:56:26.007 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:56:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:56:07.774508+0000", 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:56:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.149+0000 7f7ac2495700 1 Processor -- start 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7ac2495700 1 -- start start 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7ac2495700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc07c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7ac2495700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc07c830 con 0x7f7abc07de00 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7abbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc07c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7abbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc07c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53210/0 (socket says 172.21.15.78:53210) 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7abbfff700 1 -- 172.21.15.78:0/3901474441 learned_addr learned my addr 172.21.15.78:0/3901474441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7abbfff700 1 -- 172.21.15.78:0/3901474441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc07c9b0 con 0x7f7abc07de00 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7abbfff700 1 --2- 172.21.15.78:0/3901474441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc07c260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00098a0 tx=0x7f7ab0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732494c400b50b02 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.150+0000 7f7abaffd700 1 -- 172.21.15.78:0/3901474441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ab0004160 con 0x7f7abc07de00 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7abaffd700 1 -- 172.21.15.78:0/3901474441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f7ab0004300 con 0x7f7abc07de00 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7abaffd700 1 -- 172.21.15.78:0/3901474441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ab0011a80 con 0x7f7abc07de00 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7ac2495700 1 -- 172.21.15.78:0/3901474441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 msgr2=0x7f7abc07c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:26.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7ac2495700 1 --2- 172.21.15.78:0/3901474441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc07c260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00098a0 tx=0x7f7ab0009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7ac2495700 1 -- 172.21.15.78:0/3901474441 shutdown_connections 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7ac2495700 1 --2- 172.21.15.78:0/3901474441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc07c260 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7ac2495700 1 -- 172.21.15.78:0/3901474441 >> 172.21.15.78:0/3901474441 conn(0x7f7abc103340 msgr2=0x7f7abc105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7ac2495700 1 -- 172.21.15.78:0/3901474441 shutdown_connections 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.151+0000 7f7ac2495700 1 -- 172.21.15.78:0/3901474441 wait complete. 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7ac2495700 1 Processor -- start 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7ac2495700 1 -- start start 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7ac2495700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7ac2495700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc10e200 con 0x7f7abc07de00 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7abbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7abbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53224/0 (socket says 172.21.15.78:53224) 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7abbfff700 1 -- 172.21.15.78:0/1489091620 learned_addr learned my addr 172.21.15.78:0/1489091620 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.152+0000 7f7abbfff700 1 -- 172.21.15.78:0/1489091620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc19e250 con 0x7f7abc07de00 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.153+0000 7f7abbfff700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc19dd10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0006af0 tx=0x7f7ab0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:26.011 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.153+0000 7f7ab97fa700 1 -- 172.21.15.78:0/1489091620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ab0003930 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.153+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7abc19e4e0 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.153+0000 7f7ab97fa700 1 -- 172.21.15.78:0/1489091620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f7ab0003ad0 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.153+0000 7f7ab97fa700 1 -- 172.21.15.78:0/1489091620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ab00195e0 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.153+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7abc1a11c0 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.154+0000 7f7ab97fa700 1 -- 172.21.15.78:0/1489091620 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f7ab0019900 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.154+0000 7f7ab97fa700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f7aa4038e80 0x7f7aa403b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.154+0000 7f7ab97fa700 1 -- 172.21.15.78:0/1489091620 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7ab001a770 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.154+0000 7f7abb7fe700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f7aa4038e80 0x7f7aa403b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.154+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7abc063900 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.154+0000 7f7abb7fe700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f7aa4038e80 0x7f7aa403b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7aac006fd0 tx=0x7f7aac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.159+0000 7f7ab97fa700 1 -- 172.21.15.78:0/1489091620 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ab001aac0 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.430+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7abc1a1390 con 0x7f7abc07de00 2024-03-16T10:56:26.012 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.430+0000 7f7ab97fa700 1 -- 172.21.15.78:0/1489091620 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7ab002d960 con 0x7f7abc07de00 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f7aa4038e80 msgr2=0x7f7aa403b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f7aa4038e80 0x7f7aa403b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7aac006fd0 tx=0x7f7aac006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 msgr2=0x7f7abc19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc19dd10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0006af0 tx=0x7f7ab0011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 shutdown_connections 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f7aa4038e80 0x7f7aa403b340 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 --2- 172.21.15.78:0/1489091620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc07de00 0x7f7abc19dd10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 >> 172.21.15.78:0/1489091620 conn(0x7f7abc103340 msgr2=0x7f7abc103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 shutdown_connections 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:25.432+0000 7f7ac2495700 1 -- 172.21.15.78:0/1489091620 wait complete. 2024-03-16T10:56:26.013 INFO:teuthology.orchestra.run.smithi078.stdout:mgr is available 2024-03-16T10:56:26.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:26 smithi078 ceph-mon[32235]: mgrmap e4: x(active, since 3s) 2024-03-16T10:56:26.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:26 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1489091620' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:56:27.232 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:27 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2050855421' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T10:56:27.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T10:56:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0cff7700 1 Processor -- start 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0cff7700 1 -- start start 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0cff7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d08107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0cff7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0d08107610 con 0x7f0d08106c20 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d08107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d08107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52692/0 (socket says 172.21.15.78:52692) 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0659c700 1 -- 172.21.15.78:0/4291433915 learned_addr learned my addr 172.21.15.78:0/4291433915 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.427+0000 7f0d0659c700 1 -- 172.21.15.78:0/4291433915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d08107e20 con 0x7f0d08106c20 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0659c700 1 --2- 172.21.15.78:0/4291433915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d08107040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80098a0 tx=0x7f0cf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b7b3f69234957d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0559a700 1 -- 172.21.15.78:0/4291433915 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf8004160 con 0x7f0d08106c20 2024-03-16T10:56:27.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0559a700 1 -- 172.21.15.78:0/4291433915 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0cf8004300 con 0x7f0d08106c20 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0559a700 1 -- 172.21.15.78:0/4291433915 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf8011a80 con 0x7f0d08106c20 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0cff7700 1 -- 172.21.15.78:0/4291433915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 msgr2=0x7f0d08107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0cff7700 1 --2- 172.21.15.78:0/4291433915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d08107040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80098a0 tx=0x7f0cf8009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0cff7700 1 -- 172.21.15.78:0/4291433915 shutdown_connections 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0cff7700 1 --2- 172.21.15.78:0/4291433915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d08107040 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0cff7700 1 -- 172.21.15.78:0/4291433915 >> 172.21.15.78:0/4291433915 conn(0x7f0d08102220 msgr2=0x7f0d08104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0cff7700 1 -- 172.21.15.78:0/4291433915 shutdown_connections 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.428+0000 7f0d0cff7700 1 -- 172.21.15.78:0/4291433915 wait complete. 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.429+0000 7f0d0cff7700 1 Processor -- start 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.429+0000 7f0d0cff7700 1 -- start start 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.429+0000 7f0d0cff7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d0807ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.429+0000 7f0d0cff7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0810b050 con 0x7f0d08106c20 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.429+0000 7f0d0659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d0807ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.429+0000 7f0d0659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d0807ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52706/0 (socket says 172.21.15.78:52706) 2024-03-16T10:56:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.429+0000 7f0d0659c700 1 -- 172.21.15.78:0/2050855421 learned_addr learned my addr 172.21.15.78:0/2050855421 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.430+0000 7f0d0659c700 1 -- 172.21.15.78:0/2050855421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0819e840 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.430+0000 7f0d0659c700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d0807ba30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006af0 tx=0x7f0cf8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.430+0000 7f0cf77fe700 1 -- 172.21.15.78:0/2050855421 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf8003930 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.430+0000 7f0cf77fe700 1 -- 172.21.15.78:0/2050855421 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0cf8003ad0 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.430+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0819ea10 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.430+0000 7f0cf77fe700 1 -- 172.21.15.78:0/2050855421 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf80195e0 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.430+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0819ee30 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.431+0000 7f0cf77fe700 1 -- 172.21.15.78:0/2050855421 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f0cf80198b0 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.431+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d08063900 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.431+0000 7f0cf77fe700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f0cf0038ed0 0x7f0cf003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.431+0000 7f0cf77fe700 1 -- 172.21.15.78:0/2050855421 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0cf804d210 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.431+0000 7f0d05d9b700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f0cf0038ed0 0x7f0cf003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.432+0000 7f0d05d9b700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f0cf0038ed0 0x7f0cf003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc006fd0 tx=0x7f0cfc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.436+0000 7f0cf77fe700 1 -- 172.21.15.78:0/2050855421 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0cf800bc40 con 0x7f0d08106c20 2024-03-16T10:56:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.628+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0d081a0840 con 0x7f0d08106c20 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.629+0000 7f0cf77fe700 1 -- 172.21.15.78:0/2050855421 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+400 (secure 0 0 0) 0x7f0cf802d960 con 0x7f0d08106c20 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f0cf0038ed0 msgr2=0x7f0cf003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f0cf0038ed0 0x7f0cf003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc006fd0 tx=0x7f0cfc006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 msgr2=0x7f0d0807ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d0807ba30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006af0 tx=0x7f0cf8011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 shutdown_connections 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f0cf0038ed0 0x7f0cf003b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 --2- 172.21.15.78:0/2050855421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d08106c20 0x7f0d0807ba30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 >> 172.21.15.78:0/2050855421 conn(0x7f0d08102220 msgr2=0x7f0d08102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.631+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 shutdown_connections 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:26.632+0000 7f0d0cff7700 1 -- 172.21.15.78:0/2050855421 wait complete. 2024-03-16T10:56:27.245 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling cephadm module... 2024-03-16T10:56:28.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:28 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3255501590' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-16T10:56:28.396 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: ignoring --setuser ceph since I am not root 2024-03-16T10:56:28.396 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: ignoring --setgroup ceph since I am not root 2024-03-16T10:56:28.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.805+0000 7f5af766c700 1 Processor -- start 2024-03-16T10:56:28.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.805+0000 7f5af766c700 1 -- start start 2024-03-16T10:56:28.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.805+0000 7f5af766c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af0106400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:28.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.805+0000 7f5af766c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5af01069d0 con 0x7f5af0105fe0 2024-03-16T10:56:28.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.805+0000 7f5af5408700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af0106400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.805+0000 7f5af5408700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af0106400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52720/0 (socket says 172.21.15.78:52720) 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.805+0000 7f5af5408700 1 -- 172.21.15.78:0/3132612220 learned_addr learned my addr 172.21.15.78:0/3132612220 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5af5408700 1 -- 172.21.15.78:0/3132612220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af0108f70 con 0x7f5af0105fe0 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5af5408700 1 --2- 172.21.15.78:0/3132612220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af0106400 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0098a0 tx=0x7f5aec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4fb98b78ffbf3f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5ae7fff700 1 -- 172.21.15.78:0/3132612220 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5aec004160 con 0x7f5af0105fe0 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5ae7fff700 1 -- 172.21.15.78:0/3132612220 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5aec004300 con 0x7f5af0105fe0 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5ae7fff700 1 -- 172.21.15.78:0/3132612220 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5aec011a80 con 0x7f5af0105fe0 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5af766c700 1 -- 172.21.15.78:0/3132612220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 msgr2=0x7f5af0106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5af766c700 1 --2- 172.21.15.78:0/3132612220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af0106400 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0098a0 tx=0x7f5aec009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5af766c700 1 -- 172.21.15.78:0/3132612220 shutdown_connections 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5af766c700 1 --2- 172.21.15.78:0/3132612220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af0106400 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.806+0000 7f5af766c700 1 -- 172.21.15.78:0/3132612220 >> 172.21.15.78:0/3132612220 conn(0x7f5af01019b0 msgr2=0x7f5af0103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af766c700 1 -- 172.21.15.78:0/3132612220 shutdown_connections 2024-03-16T10:56:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af766c700 1 -- 172.21.15.78:0/3132612220 wait complete. 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af766c700 1 Processor -- start 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af766c700 1 -- start start 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af766c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af766c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5af010c1a0 con 0x7f5af0105fe0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af5408700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af5408700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af019bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52724/0 (socket says 172.21.15.78:52724) 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.807+0000 7f5af5408700 1 -- 172.21.15.78:0/3255501590 learned_addr learned my addr 172.21.15.78:0/3255501590 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.808+0000 7f5af5408700 1 -- 172.21.15.78:0/3255501590 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af019c0c0 con 0x7f5af0105fe0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.808+0000 7f5af5408700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af019bb80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006af0 tx=0x7f5aec011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.808+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5aec003930 con 0x7f5af0105fe0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.808+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5aec003ad0 con 0x7f5af0105fe0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.808+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af019c350 con 0x7f5af0105fe0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.808+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5aec0195e0 con 0x7f5af0105fe0 2024-03-16T10:56:28.689 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.808+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af019f040 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.809+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f5aec019900 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.809+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af0063900 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.809+0000 7f5ae67fc700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f5adc038e80 0x7f5adc03b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.809+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5aec04d4a0 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.809+0000 7f5af4c07700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f5adc038e80 0x7f5adc03b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.810+0000 7f5af4c07700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f5adc038e80 0x7f5adc03b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0006fd0 tx=0x7f5ae0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:27.814+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5aec04cec0 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.065+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f5af019f210 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.105+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f5af019f210 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.107+0000 7f5ae67fc700 1 -- 172.21.15.78:0/3255501590 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f5aec02d960 con 0x7f5af0105fe0 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f5adc038e80 msgr2=0x7f5adc03b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f5adc038e80 0x7f5adc03b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0006fd0 tx=0x7f5ae0006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 msgr2=0x7f5af019bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af019bb80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006af0 tx=0x7f5aec011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:28.691 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 shutdown_connections 2024-03-16T10:56:28.691 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f5adc038e80 0x7f5adc03b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:28.691 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 --2- 172.21.15.78:0/3255501590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5af0105fe0 0x7f5af019bb80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:28.691 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 >> 172.21.15.78:0/3255501590 conn(0x7f5af01019b0 msgr2=0x7f5af0102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:28.691 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 shutdown_connections 2024-03-16T10:56:28.691 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:28.108+0000 7f5af766c700 1 -- 172.21.15.78:0/3255501590 wait complete. 2024-03-16T10:56:28.998 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:28.689+0000 7fd01a578200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:56:28.999 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:28.789+0000 7fd01a578200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:56:29.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:29 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3255501590' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-16T10:56:29.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:29 smithi078 ceph-mon[32235]: mgrmap e5: x(active, since 6s) 2024-03-16T10:56:29.559 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:29.468+0000 7fd01a578200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:56:29.882 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:29.560+0000 7fd01a578200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:56:29.882 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:29.701+0000 7fd01a578200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:56:29.882 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:29.797+0000 7fd01a578200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:56:29.882 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:29.881+0000 7fd01a578200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.096+0000 7f6bba8b1700 1 Processor -- start 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.096+0000 7f6bba8b1700 1 -- start start 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.096+0000 7f6bba8b1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb41069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bba8b1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb407bd90 con 0x7f6bb41065c0 2024-03-16T10:56:29.994 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb41069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb41069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52756/0 (socket says 172.21.15.78:52756) 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb3fff700 1 -- 172.21.15.78:0/3186682805 learned_addr learned my addr 172.21.15.78:0/3186682805 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb3fff700 1 -- 172.21.15.78:0/3186682805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb4106f20 con 0x7f6bb41065c0 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb3fff700 1 --2- 172.21.15.78:0/3186682805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb41069e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40098a0 tx=0x7f6ba4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d11327693e8417 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb2ffd700 1 -- 172.21.15.78:0/3186682805 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ba4004160 con 0x7f6bb41065c0 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb2ffd700 1 -- 172.21.15.78:0/3186682805 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6ba4004300 con 0x7f6bb41065c0 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.097+0000 7f6bb2ffd700 1 -- 172.21.15.78:0/3186682805 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ba4011a80 con 0x7f6bb41065c0 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.098+0000 7f6bba8b1700 1 -- 172.21.15.78:0/3186682805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 msgr2=0x7f6bb41069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.098+0000 7f6bba8b1700 1 --2- 172.21.15.78:0/3186682805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb41069e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40098a0 tx=0x7f6ba4009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.098+0000 7f6bba8b1700 1 -- 172.21.15.78:0/3186682805 shutdown_connections 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.098+0000 7f6bba8b1700 1 --2- 172.21.15.78:0/3186682805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb41069e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.098+0000 7f6bba8b1700 1 -- 172.21.15.78:0/3186682805 >> 172.21.15.78:0/3186682805 conn(0x7f6bb4102180 msgr2=0x7f6bb41045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.098+0000 7f6bba8b1700 1 -- 172.21.15.78:0/3186682805 shutdown_connections 2024-03-16T10:56:29.995 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.098+0000 7f6bba8b1700 1 -- 172.21.15.78:0/3186682805 wait complete. 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bba8b1700 1 Processor -- start 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bba8b1700 1 -- start start 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bba8b1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bba8b1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb4107e70 con 0x7f6bb41065c0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bb3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bb3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52760/0 (socket says 172.21.15.78:52760) 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bb3fff700 1 -- 172.21.15.78:0/2826432054 learned_addr learned my addr 172.21.15.78:0/2826432054 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bb3fff700 1 -- 172.21.15.78:0/2826432054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb419e270 con 0x7f6bb41065c0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.099+0000 7f6bb3fff700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb419dd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4006af0 tx=0x7f6ba4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.100+0000 7f6bb17fa700 1 -- 172.21.15.78:0/2826432054 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ba40038b0 con 0x7f6bb41065c0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.100+0000 7f6bb17fa700 1 -- 172.21.15.78:0/2826432054 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6ba4003a50 con 0x7f6bb41065c0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.100+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb419e500 con 0x7f6bb41065c0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.100+0000 7f6bb17fa700 1 -- 172.21.15.78:0/2826432054 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ba4019550 con 0x7f6bb41065c0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.100+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb41a11f0 con 0x7f6bb41065c0 2024-03-16T10:56:29.996 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.100+0000 7f6bb17fa700 1 -- 172.21.15.78:0/2826432054 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f6ba4019800 con 0x7f6bb41065c0 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.101+0000 7f6bb17fa700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 0x7f6b9c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.101+0000 7f6bb17fa700 1 -- 172.21.15.78:0/2826432054 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6ba404d0f0 con 0x7f6bb41065c0 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.101+0000 7f6bb37fe700 1 -- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 msgr2=0x7f6b9c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2003716687 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.101+0000 7f6bb37fe700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 0x7f6b9c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.101+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb4063900 con 0x7f6bb41065c0 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.106+0000 7f6bb17fa700 1 -- 172.21.15.78:0/2826432054 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ba4003c00 con 0x7f6bb41065c0 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.302+0000 7f6bb37fe700 1 -- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 msgr2=0x7f6b9c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2003716687 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.302+0000 7f6bb37fe700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 0x7f6b9c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.382+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f6bb402dd00 con 0x7f6bb41065c0 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.383+0000 7f6bb17fa700 1 -- 172.21.15.78:0/2826432054 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f6ba402d960 con 0x7f6bb41065c0 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 msgr2=0x7f6b9c03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 0x7f6b9c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 msgr2=0x7f6bb419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:29.997 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb419dd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4006af0 tx=0x7f6ba4011b00 comp rx=0 tx=0).stop 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 shutdown_connections 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7f6b9c038f20 0x7f6b9c03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 --2- 172.21.15.78:0/2826432054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb41065c0 0x7f6bb419dd30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.386+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 >> 172.21.15.78:0/2826432054 conn(0x7f6bb4102180 msgr2=0x7f6bb4194a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.387+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 shutdown_connections 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:29.387+0000 7f6bba8b1700 1 -- 172.21.15.78:0/2826432054 wait complete. 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-03-16T10:56:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 5... 2024-03-16T10:56:30.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:30 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2826432054' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T10:56:30.421 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:30.154+0000 7fd01a578200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:56:30.422 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:30.266+0000 7fd01a578200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:56:30.422 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:30.353+0000 7fd01a578200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:56:30.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:30.453+0000 7fd01a578200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:56:30.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:30.625+0000 7fd01a578200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:56:31.829 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:31.576+0000 7fd01a578200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:56:31.829 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:31.704+0000 7fd01a578200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:56:32.121 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:32.032+0000 7fd01a578200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:56:32.500 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:32.120+0000 7fd01a578200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:56:32.500 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:32.315+0000 7fd01a578200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:56:32.500 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:32.499+0000 7fd01a578200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:56:33.043 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:32.782+0000 7fd01a578200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:56:33.063 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:33 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:33.042+0000 7fd01a578200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:56:33.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:33 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:33.276+0000 7fd01a578200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:56:34.559 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:34.069+0000 7fd01a578200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:56:34.947 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:34.635+0000 7fd01a578200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:56:34.947 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:34.732+0000 7fd01a578200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:56:34.947 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:34.946+0000 7fd01a578200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:56:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: Active manager daemon x restarted 2024-03-16T10:56:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: Activating manager daemon x 2024-03-16T10:56:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: osdmap e2: 0 total, 0 up, 0 in 2024-03-16T10:56:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: mgrmap e6: x(active, starting, since 0.00339785s) 2024-03-16T10:56:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: Manager daemon x is now available 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:56:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:35 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:56:36.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:36 smithi078 ceph-mon[32235]: Found migration_current of "None". Setting to last migration. 2024-03-16T10:56:36.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:36 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:56:36.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:36 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T10:56:36.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:36 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T10:56:36.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:36 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:36.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:36 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:36.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:36 smithi078 ceph-mon[32235]: mgrmap e7: x(active, since 1.0061s) 2024-03-16T10:56:36.519 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:56:36.519 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-16T10:56:36.519 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T10:56:36.519 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:56:36.519 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.420+0000 7ff40a28e700 1 Processor -- start 2024-03-16T10:56:36.519 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.420+0000 7ff40a28e700 1 -- start start 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff40a28e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40410ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff40a28e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff4040635f0 con 0x7ff404108760 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff403fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40410ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff403fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40410ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52762/0 (socket says 172.21.15.78:52762) 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff403fff700 1 -- 172.21.15.78:0/292368321 learned_addr learned my addr 172.21.15.78:0/292368321 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff403fff700 1 -- 172.21.15.78:0/292368321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40410b090 con 0x7ff404108760 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff403fff700 1 --2- 172.21.15.78:0/292368321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40410ab50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40098a0 tx=0x7ff3f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51157d8def774ed1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.421+0000 7ff402ffd700 1 -- 172.21.15.78:0/292368321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3f4004160 con 0x7ff404108760 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff402ffd700 1 -- 172.21.15.78:0/292368321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff3f4004300 con 0x7ff404108760 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff402ffd700 1 -- 172.21.15.78:0/292368321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3f4011a80 con 0x7ff404108760 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff40a28e700 1 -- 172.21.15.78:0/292368321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 msgr2=0x7ff40410ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff40a28e700 1 --2- 172.21.15.78:0/292368321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40410ab50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40098a0 tx=0x7ff3f4009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff40a28e700 1 -- 172.21.15.78:0/292368321 shutdown_connections 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff40a28e700 1 --2- 172.21.15.78:0/292368321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40410ab50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff40a28e700 1 -- 172.21.15.78:0/292368321 >> 172.21.15.78:0/292368321 conn(0x7ff4041021b0 msgr2=0x7ff4041045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff40a28e700 1 -- 172.21.15.78:0/292368321 shutdown_connections 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.422+0000 7ff40a28e700 1 -- 172.21.15.78:0/292368321 wait complete. 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.423+0000 7ff40a28e700 1 Processor -- start 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.423+0000 7ff40a28e700 1 -- start start 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.423+0000 7ff40a28e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40419bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.423+0000 7ff40a28e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff40410bf80 con 0x7ff404108760 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.423+0000 7ff403fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40419bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.423+0000 7ff403fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40419bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52772/0 (socket says 172.21.15.78:52772) 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.423+0000 7ff403fff700 1 -- 172.21.15.78:0/769511956 learned_addr learned my addr 172.21.15.78:0/769511956 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.424+0000 7ff403fff700 1 -- 172.21.15.78:0/769511956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40419c050 con 0x7ff404108760 2024-03-16T10:56:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.424+0000 7ff403fff700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40419bb10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4011a50 tx=0x7ff3f4004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.424+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3f40039f0 con 0x7ff404108760 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.424+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff3f4003b90 con 0x7ff404108760 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.424+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40419c2e0 con 0x7ff404108760 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.425+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3f401a600 con 0x7ff404108760 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.425+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40419efd0 con 0x7ff404108760 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.425+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7ff3f401a980 con 0x7ff404108760 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.425+0000 7ff4017fa700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.425+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff3f404fb60 con 0x7ff404108760 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.425+0000 7ff4037fe700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 msgr2=0x7ff3ec03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2003716687 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.425+0000 7ff4037fe700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.426+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 --> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff404063900 con 0x7ff3ec038f70 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.626+0000 7ff4037fe700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 msgr2=0x7ff3ec03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2003716687 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:30.626+0000 7ff4037fe700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:56:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:31.027+0000 7ff4037fe700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 msgr2=0x7ff3ec03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2003716687 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:31.027+0000 7ff4037fe700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:31.828+0000 7ff4037fe700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 msgr2=0x7ff3ec03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2003716687 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:31.828+0000 7ff4037fe700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:33.429+0000 7ff4037fe700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 msgr2=0x7ff3ec03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2003716687 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:33.429+0000 7ff4037fe700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:34.962+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7ff3f4024a40 con 0x7ff404108760 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:34.962+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 msgr2=0x7ff3ec03b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:34.962+0000 7ff4017fa700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.963+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7ff3f4041fc0 con 0x7ff404108760 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.964+0000 7ff4017fa700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7ff3ec03c670 0x7ff3ec03ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.964+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff3f4012ea0 con 0x7ff3ec03c670 2024-03-16T10:56:36.523 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.964+0000 7ff4037fe700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7ff3ec03c670 0x7ff3ec03ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.965+0000 7ff4037fe700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7ff3ec03c670 0x7ff3ec03ea60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8003a10 tx=0x7ff3f80092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.966+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff3f4012ea0 con 0x7ff3ec03c670 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.977+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff4041034c0 con 0x7ff3ec03c670 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.977+0000 7ff4017fa700 1 -- 172.21.15.78:0/769511956 <== mgr.14120 v2:172.21.15.78:6800/3269664941 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff4041034c0 con 0x7ff3ec03c670 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7ff3ec03c670 msgr2=0x7ff3ec03ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7ff3ec03c670 0x7ff3ec03ea60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8003a10 tx=0x7ff3f80092b0 comp rx=0 tx=0).stop 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 msgr2=0x7ff40419bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40419bb10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4011a50 tx=0x7ff3f4004290 comp rx=0 tx=0).stop 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 shutdown_connections 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7ff3ec03c670 0x7ff3ec03ea60 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff404108760 0x7ff40419bb10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 --2- 172.21.15.78:0/769511956 >> [v2:172.21.15.78:6800/2003716687,v1:172.21.15.78:6801/2003716687] conn(0x7ff3ec038f70 0x7ff3ec03b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 >> 172.21.15.78:0/769511956 conn(0x7ff4041021b0 msgr2=0x7ff404102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:36.525 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 shutdown_connections 2024-03-16T10:56:36.525 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:35.978+0000 7ff40a28e700 1 -- 172.21.15.78:0/769511956 wait complete. 2024-03-16T10:56:36.525 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 5 is available 2024-03-16T10:56:36.525 INFO:teuthology.orchestra.run.smithi078.stdout:Setting orchestrator backend to cephadm... 2024-03-16T10:56:37.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:37 smithi078 ceph-mon[32235]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T10:56:37.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:37 smithi078 ceph-mon[32235]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T10:56:37.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:37 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:56:37.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.001+0000 7f5a04a1c700 1 Processor -- start 2024-03-16T10:56:37.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.001+0000 7f5a04a1c700 1 -- start start 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.001+0000 7f5a04a1c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a00107010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.001+0000 7f5a04a1c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a001075e0 con 0x7f5a00106bf0 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fe59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a00107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fe59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a00107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35720/0 (socket says 172.21.15.78:35720) 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fe59c700 1 -- 172.21.15.78:0/3789232090 learned_addr learned my addr 172.21.15.78:0/3789232090 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fe59c700 1 -- 172.21.15.78:0/3789232090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a00107df0 con 0x7f5a00106bf0 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fe59c700 1 --2- 172.21.15.78:0/3789232090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a00107010 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009e80 tx=0x7f59f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcf09cae3dc7a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fd59a700 1 -- 172.21.15.78:0/3789232090 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59f0017070 con 0x7f5a00106bf0 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fd59a700 1 -- 172.21.15.78:0/3789232090 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f59f000b8c0 con 0x7f5a00106bf0 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.002+0000 7f59fd59a700 1 -- 172.21.15.78:0/3789232090 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59f0004520 con 0x7f5a00106bf0 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.003+0000 7f5a04a1c700 1 -- 172.21.15.78:0/3789232090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 msgr2=0x7f5a00107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:37.807 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.003+0000 7f5a04a1c700 1 --2- 172.21.15.78:0/3789232090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a00107010 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009e80 tx=0x7f59f000b330 comp rx=0 tx=0).stop 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.003+0000 7f5a04a1c700 1 -- 172.21.15.78:0/3789232090 shutdown_connections 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.003+0000 7f5a04a1c700 1 --2- 172.21.15.78:0/3789232090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a00107010 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.003+0000 7f5a04a1c700 1 -- 172.21.15.78:0/3789232090 >> 172.21.15.78:0/3789232090 conn(0x7f5a001021b0 msgr2=0x7f5a001045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.003+0000 7f5a04a1c700 1 -- 172.21.15.78:0/3789232090 shutdown_connections 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.003+0000 7f5a04a1c700 1 -- 172.21.15.78:0/3789232090 wait complete. 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f5a04a1c700 1 Processor -- start 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f5a04a1c700 1 -- start start 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f5a04a1c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a000631f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f5a04a1c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0010b020 con 0x7f5a00106bf0 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f59fe59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a000631f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f59fe59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a000631f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35732/0 (socket says 172.21.15.78:35732) 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f59fe59c700 1 -- 172.21.15.78:0/1985457299 learned_addr learned my addr 172.21.15.78:0/1985457299 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:37.808 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f59fe59c700 1 -- 172.21.15.78:0/1985457299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0019e800 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f59fe59c700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a000631f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f59f0000f80 tx=0x7f59f0009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.004+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59f0020020 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.005+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0019e9d0 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.005+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f59f000bba0 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.005+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59f0017040 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.005+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0019ee50 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.005+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f59f0023020 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.005+0000 7f59f77fe700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f59ec038e50 0x7f59ec03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.005+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a00063900 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.006+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f59f000f050 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.006+0000 7f59fdd9b700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f59ec038e50 0x7f59ec03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.006+0000 7f59fdd9b700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f59ec038e50 0x7f59ec03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59e8006fd0 tx=0x7f59e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.011+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59f002a020 con 0x7f5a00106bf0 2024-03-16T10:56:37.809 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.015+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f59f0025400 con 0x7f5a00106bf0 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.221+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f5a00103420 con 0x7f59ec038e50 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.228+0000 7f59f77fe700 1 -- 172.21.15.78:0/1985457299 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5a00103420 con 0x7f59ec038e50 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f59ec038e50 msgr2=0x7f59ec03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f59ec038e50 0x7f59ec03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59e8006fd0 tx=0x7f59e8006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 msgr2=0x7f5a000631f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a000631f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f59f0000f80 tx=0x7f59f0009640 comp rx=0 tx=0).stop 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 shutdown_connections 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f59ec038e50 0x7f59ec03b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 --2- 172.21.15.78:0/1985457299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a00106bf0 0x7f5a000631f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 >> 172.21.15.78:0/1985457299 conn(0x7f5a001021b0 msgr2=0x7f5a00102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 shutdown_connections 2024-03-16T10:56:37.810 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:37.230+0000 7f5a04a1c700 1 -- 172.21.15.78:0/1985457299 wait complete. 2024-03-16T10:56:38.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:38 smithi078 ceph-mon[32235]: [16/Mar/2024:10:56:36] ENGINE Bus STARTING 2024-03-16T10:56:38.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:38 smithi078 ceph-mon[32235]: [16/Mar/2024:10:56:36] ENGINE Serving on https://172.21.15.78:7150 2024-03-16T10:56:38.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:38 smithi078 ceph-mon[32235]: [16/Mar/2024:10:56:36] ENGINE Serving on http://172.21.15.78:8765 2024-03-16T10:56:38.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:38 smithi078 ceph-mon[32235]: [16/Mar/2024:10:56:36] ENGINE Bus STARTED 2024-03-16T10:56:38.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:38 smithi078 ceph-mon[32235]: mgrmap e8: x(active, since 2s) 2024-03-16T10:56:38.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:38 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:38.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:38 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:56:39.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-mon[32235]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:56:39.070 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdcd5ea3700 1 Processor -- start 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdcd5ea3700 1 -- start start 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdcd5ea3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd0108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdcd5ea3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd01086f0 con 0x7fdcd0107d00 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdccf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd0108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdccf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd0108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35744/0 (socket says 172.21.15.78:35744) 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdccf7fe700 1 -- 172.21.15.78:0/2550006753 learned_addr learned my addr 172.21.15.78:0/2550006753 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdccf7fe700 1 -- 172.21.15.78:0/2550006753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd0108f00 con 0x7fdcd0107d00 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdccf7fe700 1 --2- 172.21.15.78:0/2550006753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd0108120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00098a0 tx=0x7fdcc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed448271a4918ed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.250+0000 7fdcce7fc700 1 -- 172.21.15.78:0/2550006753 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcc0004160 con 0x7fdcd0107d00 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcce7fc700 1 -- 172.21.15.78:0/2550006753 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdcc0004300 con 0x7fdcd0107d00 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcce7fc700 1 -- 172.21.15.78:0/2550006753 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcc0011a80 con 0x7fdcd0107d00 2024-03-16T10:56:39.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/2550006753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 msgr2=0x7fdcd0108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcd5ea3700 1 --2- 172.21.15.78:0/2550006753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd0108120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00098a0 tx=0x7fdcc0009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/2550006753 shutdown_connections 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcd5ea3700 1 --2- 172.21.15.78:0/2550006753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd0108120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/2550006753 >> 172.21.15.78:0/2550006753 conn(0x7fdcd0103340 msgr2=0x7fdcd0105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/2550006753 shutdown_connections 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.251+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/2550006753 wait complete. 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdcd5ea3700 1 Processor -- start 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdcd5ea3700 1 -- start start 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdcd5ea3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdcd5ea3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd010c130 con 0x7fdcd0107d00 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdccf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdccf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35748/0 (socket says 172.21.15.78:35748) 2024-03-16T10:56:39.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdccf7fe700 1 -- 172.21.15.78:0/381509545 learned_addr learned my addr 172.21.15.78:0/381509545 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.252+0000 7fdccf7fe700 1 -- 172.21.15.78:0/381509545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd019c0b0 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.253+0000 7fdccf7fe700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd019bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0006af0 tx=0x7fdcc0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.253+0000 7fdcccff9700 1 -- 172.21.15.78:0/381509545 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcc0003930 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.253+0000 7fdcccff9700 1 -- 172.21.15.78:0/381509545 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdcc0003ad0 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.253+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd019c340 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.253+0000 7fdcccff9700 1 -- 172.21.15.78:0/381509545 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcc00195e0 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.253+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd019f030 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.254+0000 7fdcccff9700 1 -- 172.21.15.78:0/381509545 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fdcc0019910 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.254+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd0063900 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.254+0000 7fdcccff9700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fdcb8038f20 0x7fdcb803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.254+0000 7fdcccff9700 1 -- 172.21.15.78:0/381509545 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdcc004d560 con 0x7fdcd0107d00 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.254+0000 7fdcceffd700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fdcb8038f20 0x7fdcb803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.254+0000 7fdcceffd700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fdcb8038f20 0x7fdcb803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4006fd0 tx=0x7fdcc4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:39.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.259+0000 7fdcccff9700 1 -- 172.21.15.78:0/381509545 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdcc0019d00 con 0x7fdcd0107d00 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.454+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fdcd01045e0 con 0x7fdcb8038f20 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.455+0000 7fdcccff9700 1 -- 172.21.15.78:0/381509545 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fdcd01045e0 con 0x7fdcb8038f20 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fdcb8038f20 msgr2=0x7fdcb803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fdcb8038f20 0x7fdcb803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4006fd0 tx=0x7fdcc4006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 msgr2=0x7fdcd019bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd019bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0006af0 tx=0x7fdcc0011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 shutdown_connections 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fdcb8038f20 0x7fdcb803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 --2- 172.21.15.78:0/381509545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcd0107d00 0x7fdcd019bb70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 >> 172.21.15.78:0/381509545 conn(0x7fdcd0103340 msgr2=0x7fdcd0103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 shutdown_connections 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:38.457+0000 7fdcd5ea3700 1 -- 172.21.15.78:0/381509545 wait complete. 2024-03-16T10:56:39.074 INFO:teuthology.orchestra.run.smithi078.stdout:Generating ssh key... 2024-03-16T10:56:40.020 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: Generating public/private rsa key pair. 2024-03-16T10:56:40.020 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: Your identification has been saved in /tmp/tmpizca0mqr/key. 2024-03-16T10:56:40.020 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: Your public key has been saved in /tmp/tmpizca0mqr/key.pub. 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: The key fingerprint is: 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: SHA256:0fabJsKF+3WfIsRuqVygOCdiggF3s4YU4zb+9ZYxppc ceph-a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: The key's randomart image is: 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: +---[RSA 3072]----+ 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: | o | 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: | . o . | 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: |. * o . o | 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: |.= + o + . | 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: |. o o . S.o . | 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: |.. o ..=.B.o o | 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: |.. o.+.oE +.* . | 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: | o . +o.o.O o ..| 2024-03-16T10:56:40.021 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: | o+ . ...| 2024-03-16T10:56:40.022 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: +----[SHA256]-----+ 2024-03-16T10:56:40.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:40 smithi078 ceph-mon[32235]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:56:40.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:40 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:40.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:40 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:40.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.482+0000 7f70d84ae700 1 Processor -- start 2024-03-16T10:56:40.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.482+0000 7f70d84ae700 1 -- start start 2024-03-16T10:56:40.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.482+0000 7f70d84ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d0108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:40.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.482+0000 7f70d84ae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01086f0 con 0x7f70d0107d00 2024-03-16T10:56:40.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.482+0000 7f70d624a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d0108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:40.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.482+0000 7f70d624a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d0108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35760/0 (socket says 172.21.15.78:35760) 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.482+0000 7f70d624a700 1 -- 172.21.15.78:0/3094003080 learned_addr learned my addr 172.21.15.78:0/3094003080 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.483+0000 7f70d624a700 1 -- 172.21.15.78:0/3094003080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d0108f00 con 0x7f70d0107d00 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.483+0000 7f70d624a700 1 --2- 172.21.15.78:0/3094003080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d0108120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0098a0 tx=0x7f70cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea1f136393efe54 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.483+0000 7f70d5248700 1 -- 172.21.15.78:0/3094003080 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc004160 con 0x7f70d0107d00 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.483+0000 7f70d5248700 1 -- 172.21.15.78:0/3094003080 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f70cc004300 con 0x7f70d0107d00 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.483+0000 7f70d5248700 1 -- 172.21.15.78:0/3094003080 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc011a80 con 0x7f70d0107d00 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.483+0000 7f70d84ae700 1 -- 172.21.15.78:0/3094003080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 msgr2=0x7f70d0108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.483+0000 7f70d84ae700 1 --2- 172.21.15.78:0/3094003080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d0108120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0098a0 tx=0x7f70cc009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 -- 172.21.15.78:0/3094003080 shutdown_connections 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 --2- 172.21.15.78:0/3094003080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d0108120 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 -- 172.21.15.78:0/3094003080 >> 172.21.15.78:0/3094003080 conn(0x7f70d0103340 msgr2=0x7f70d0105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 -- 172.21.15.78:0/3094003080 shutdown_connections 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 -- 172.21.15.78:0/3094003080 wait complete. 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 Processor -- start 2024-03-16T10:56:40.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 -- start start 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d84ae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f70d010c130 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d624a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d624a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35762/0 (socket says 172.21.15.78:35762) 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.484+0000 7f70d624a700 1 -- 172.21.15.78:0/1196901430 learned_addr learned my addr 172.21.15.78:0/1196901430 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.485+0000 7f70d624a700 1 -- 172.21.15.78:0/1196901430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d019c0b0 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.485+0000 7f70d624a700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d019bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70cc006af0 tx=0x7f70cc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.485+0000 7f70c77fe700 1 -- 172.21.15.78:0/1196901430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc003930 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.485+0000 7f70c77fe700 1 -- 172.21.15.78:0/1196901430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f70cc003ad0 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.485+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d019c340 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.485+0000 7f70c77fe700 1 -- 172.21.15.78:0/1196901430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc0195e0 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.485+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d019f030 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.486+0000 7f70c77fe700 1 -- 172.21.15.78:0/1196901430 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f70cc019890 con 0x7f70d0107d00 2024-03-16T10:56:40.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.486+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d0063900 con 0x7f70d0107d00 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.486+0000 7f70c77fe700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f70bc038f20 0x7f70bc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.486+0000 7f70c77fe700 1 -- 172.21.15.78:0/1196901430 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f70cc04d330 con 0x7f70d0107d00 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.486+0000 7f70d5a49700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f70bc038f20 0x7f70bc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.487+0000 7f70d5a49700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f70bc038f20 0x7f70bc03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70c0006fd0 tx=0x7f70c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.491+0000 7f70c77fe700 1 -- 172.21.15.78:0/1196901430 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f70cc00bc40 con 0x7f70d0107d00 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.684+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f70d01045e0 con 0x7f70bc038f20 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.830+0000 7f70c77fe700 1 -- 172.21.15.78:0/1196901430 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f70d01045e0 con 0x7f70bc038f20 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f70bc038f20 msgr2=0x7f70bc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f70bc038f20 0x7f70bc03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70c0006fd0 tx=0x7f70c0006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 msgr2=0x7f70d019bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d019bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70cc006af0 tx=0x7f70cc011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 shutdown_connections 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f70bc038f20 0x7f70bc03b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 --2- 172.21.15.78:0/1196901430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d0107d00 0x7f70d019bb70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:40.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.832+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 >> 172.21.15.78:0/1196901430 conn(0x7f70d0103340 msgr2=0x7f70d0103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:40.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.833+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 shutdown_connections 2024-03-16T10:56:40.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:39.833+0000 7f70d84ae700 1 -- 172.21.15.78:0/1196901430 wait complete. 2024-03-16T10:56:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:41 smithi078 ceph-mon[32235]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:56:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:41 smithi078 ceph-mon[32235]: Generating ssh key... 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7DORGCym24lbxU4CZMZxZiOeHQgz0UZt44eZjronrJjH3i8WiBdc5wcPPo1m3lZdoBqqy34ZNVi2wcrqubJY6P3HGewYrvtykXnlG5GdaPlvQ3bxvVUR0Bm9BgSJ4pwnMiEosb3AforEluAVn2HnGWrtzJ44GLvcC06jh+T4bgEQ9rFpYzLD7ZbJ+Sv+MN0gkR4RZBK6QTm1i97xgIeCk19qX+Nl/BjubDZti0F0eHZkfxzJHM4HORBcnbSiLa1FrKncWVHUDQhGsZEUzN766HHWJG92ad/p+NbSEp2HVa9ovSaQm+yGJvTOUIXJIGY+pmVv06TJfUCD6qyPx5X4r9DHAZkWlrCT6JzWP2OuHi+BrIHKhsSGbn2G0Tx960pIq3dDSLZkPSbcz9DpMl2coJVgITi34mcQUEpi1TzMC9X36hdBpi+WhP/wXJjPF0Tvm+bxBB3ftCqsyVPoE2Z5nU6CYGSrVHPKHBFepF3l43NPVmEtcekwmDZHXGfvNTac= ceph-a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.836+0000 7f547ad00700 1 Processor -- start 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.836+0000 7f547ad00700 1 -- start start 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.836+0000 7f547ad00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f5474108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.836+0000 7f547ad00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f54741086f0 con 0x7f5474107d00 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.836+0000 7f5478a9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f5474108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.837+0000 7f5478a9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f5474108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35770/0 (socket says 172.21.15.78:35770) 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.837+0000 7f5478a9c700 1 -- 172.21.15.78:0/1559449811 learned_addr learned my addr 172.21.15.78:0/1559449811 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.837+0000 7f5478a9c700 1 -- 172.21.15.78:0/1559449811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5474108f00 con 0x7f5474107d00 2024-03-16T10:56:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.837+0000 7f5478a9c700 1 --2- 172.21.15.78:0/1559449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f5474108120 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f54680098a0 tx=0x7f5468009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd80957fd3af2ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.837+0000 7f54737fe700 1 -- 172.21.15.78:0/1559449811 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5468004160 con 0x7f5474107d00 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.837+0000 7f54737fe700 1 -- 172.21.15.78:0/1559449811 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5468004300 con 0x7f5474107d00 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.837+0000 7f54737fe700 1 -- 172.21.15.78:0/1559449811 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5468011a80 con 0x7f5474107d00 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 -- 172.21.15.78:0/1559449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 msgr2=0x7f5474108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 --2- 172.21.15.78:0/1559449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f5474108120 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f54680098a0 tx=0x7f5468009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 -- 172.21.15.78:0/1559449811 shutdown_connections 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 --2- 172.21.15.78:0/1559449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f5474108120 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 -- 172.21.15.78:0/1559449811 >> 172.21.15.78:0/1559449811 conn(0x7f5474103340 msgr2=0x7f5474105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 -- 172.21.15.78:0/1559449811 shutdown_connections 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 -- 172.21.15.78:0/1559449811 wait complete. 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 Processor -- start 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 -- start start 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f547419bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.838+0000 7f547ad00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f547410c130 con 0x7f5474107d00 2024-03-16T10:56:41.668 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5478a9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f547419bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5478a9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f547419bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35782/0 (socket says 172.21.15.78:35782) 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5478a9c700 1 -- 172.21.15.78:0/1285057261 learned_addr learned my addr 172.21.15.78:0/1285057261 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5478a9c700 1 -- 172.21.15.78:0/1285057261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547419c0b0 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5478a9c700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f547419bb70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5468006af0 tx=0x7f5468011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5471ffb700 1 -- 172.21.15.78:0/1285057261 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5468003930 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5471ffb700 1 -- 172.21.15.78:0/1285057261 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5468003ad0 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547419c340 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f5471ffb700 1 -- 172.21.15.78:0/1285057261 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54680195e0 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.839+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f547419f030 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.840+0000 7f5471ffb700 1 -- 172.21.15.78:0/1285057261 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f54680198e0 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.840+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5474063900 con 0x7f5474107d00 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.840+0000 7f5471ffb700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5464038ed0 0x7f546403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:41.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.840+0000 7f5471ffb700 1 -- 172.21.15.78:0/1285057261 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f546804d410 con 0x7f5474107d00 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.841+0000 7f5473fff700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5464038ed0 0x7f546403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.841+0000 7f5473fff700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5464038ed0 0x7f546403b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5460006fd0 tx=0x7f5460006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:40.845+0000 7f5471ffb700 1 -- 172.21.15.78:0/1285057261 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f546804cd60 con 0x7f5474107d00 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.043+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f54741045e0 con 0x7f5464038ed0 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.044+0000 7f5471ffb700 1 -- 172.21.15.78:0/1285057261 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f54741045e0 con 0x7f5464038ed0 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5464038ed0 msgr2=0x7f546403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5464038ed0 0x7f546403b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5460006fd0 tx=0x7f5460006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 msgr2=0x7f547419bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f547419bb70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5468006af0 tx=0x7f5468011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 shutdown_connections 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5464038ed0 0x7f546403b390 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 --2- 172.21.15.78:0/1285057261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5474107d00 0x7f547419bb70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 >> 172.21.15.78:0/1285057261 conn(0x7f5474103340 msgr2=0x7f5474103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 shutdown_connections 2024-03-16T10:56:41.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:41.046+0000 7f547ad00700 1 -- 172.21.15.78:0/1285057261 wait complete. 2024-03-16T10:56:41.671 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-16T10:56:41.671 INFO:teuthology.orchestra.run.smithi078.stdout:Adding key to root@localhost authorized_keys... 2024-03-16T10:56:41.671 INFO:teuthology.orchestra.run.smithi078.stdout:Adding host smithi078... 2024-03-16T10:56:42.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:42 smithi078 ceph-mon[32235]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:56:43.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:43 smithi078 ceph-mon[32235]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:56:43.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:43 smithi078 ceph-mon[32235]: Deploying cephadm binary to smithi078 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Added host 'smithi078' with addr '172.21.15.78' 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea90c35700 1 Processor -- start 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea90c35700 1 -- start start 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea90c35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c07c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea90c35700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c07c830 con 0x7fea8c07de00 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea8a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c07c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea8a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c07c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35786/0 (socket says 172.21.15.78:35786) 2024-03-16T10:56:47.125 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea8a59c700 1 -- 172.21.15.78:0/3943006382 learned_addr learned my addr 172.21.15.78:0/3943006382 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.035+0000 7fea8a59c700 1 -- 172.21.15.78:0/3943006382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c07c9b0 con 0x7fea8c07de00 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea8a59c700 1 --2- 172.21.15.78:0/3943006382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c07c260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0098a0 tx=0x7fea7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ab2cfafb832f46 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea8959a700 1 -- 172.21.15.78:0/3943006382 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea7c004160 con 0x7fea8c07de00 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea8959a700 1 -- 172.21.15.78:0/3943006382 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fea7c004300 con 0x7fea8c07de00 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea8959a700 1 -- 172.21.15.78:0/3943006382 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea7c011a80 con 0x7fea8c07de00 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea90c35700 1 -- 172.21.15.78:0/3943006382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 msgr2=0x7fea8c07c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea90c35700 1 --2- 172.21.15.78:0/3943006382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c07c260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0098a0 tx=0x7fea7c009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea90c35700 1 -- 172.21.15.78:0/3943006382 shutdown_connections 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea90c35700 1 --2- 172.21.15.78:0/3943006382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c07c260 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.036+0000 7fea90c35700 1 -- 172.21.15.78:0/3943006382 >> 172.21.15.78:0/3943006382 conn(0x7fea8c1032c0 msgr2=0x7fea8c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea90c35700 1 -- 172.21.15.78:0/3943006382 shutdown_connections 2024-03-16T10:56:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea90c35700 1 -- 172.21.15.78:0/3943006382 wait complete. 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea90c35700 1 Processor -- start 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea90c35700 1 -- start start 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea90c35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea90c35700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c10e1c0 con 0x7fea8c07de00 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea8a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.037+0000 7fea8a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c19dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35794/0 (socket says 172.21.15.78:35794) 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea8a59c700 1 -- 172.21.15.78:0/1381561469 learned_addr learned my addr 172.21.15.78:0/1381561469 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea8a59c700 1 -- 172.21.15.78:0/1381561469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c19e210 con 0x7fea8c07de00 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea8a59c700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c19dcd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea7c006af0 tx=0x7fea7c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea7b7fe700 1 -- 172.21.15.78:0/1381561469 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea7c0039e0 con 0x7fea8c07de00 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea7b7fe700 1 -- 172.21.15.78:0/1381561469 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fea7c003b80 con 0x7fea8c07de00 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8c19e4a0 con 0x7fea8c07de00 2024-03-16T10:56:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea7b7fe700 1 -- 172.21.15.78:0/1381561469 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea7c0196a0 con 0x7fea8c07de00 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.038+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8c1a1180 con 0x7fea8c07de00 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.039+0000 7fea7b7fe700 1 -- 172.21.15.78:0/1381561469 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fea7c019950 con 0x7fea8c07de00 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.039+0000 7fea7b7fe700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fea74038f20 0x7fea7403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.039+0000 7fea7b7fe700 1 -- 172.21.15.78:0/1381561469 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fea7c01a840 con 0x7fea8c07de00 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.039+0000 7fea89d9b700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fea74038f20 0x7fea7403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.039+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea8c063900 con 0x7fea8c07de00 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.040+0000 7fea89d9b700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fea74038f20 0x7fea7403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fea80006fd0 tx=0x7fea80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.044+0000 7fea7b7fe700 1 -- 172.21.15.78:0/1381561469 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea7c02d960 con 0x7fea8c07de00 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:42.240+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}) -- 0x7fea8c104580 con 0x7fea74038f20 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.492+0000 7fea7b7fe700 1 -- 172.21.15.78:0/1381561469 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fea8c104580 con 0x7fea74038f20 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fea74038f20 msgr2=0x7fea7403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fea74038f20 0x7fea7403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fea80006fd0 tx=0x7fea80006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 msgr2=0x7fea8c19dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:47.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c19dcd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea7c006af0 tx=0x7fea7c011c40 comp rx=0 tx=0).stop 2024-03-16T10:56:47.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 shutdown_connections 2024-03-16T10:56:47.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7fea74038f20 0x7fea7403b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:47.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 --2- 172.21.15.78:0/1381561469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea8c07de00 0x7fea8c19dcd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:47.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 >> 172.21.15.78:0/1381561469 conn(0x7fea8c1032c0 msgr2=0x7fea8c103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:47.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 shutdown_connections 2024-03-16T10:56:47.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:46.494+0000 7fea90c35700 1 -- 172.21.15.78:0/1381561469 wait complete. 2024-03-16T10:56:47.129 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mon service... 2024-03-16T10:56:47.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:47 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:47.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:47 smithi078 ceph-mon[32235]: Added host smithi078 2024-03-16T10:56:47.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:47 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:56:49.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:48 smithi078 ceph-mon[32235]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:56:49.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:48 smithi078 ceph-mon[32235]: Saving service mon spec with placement count:5 2024-03-16T10:56:49.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:48 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:49.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2aced18700 1 Processor -- start 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2aced18700 1 -- start start 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2aced18700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac81080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2aced18700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac81086b0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2accab4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac81080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2accab4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac81080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55772/0 (socket says 172.21.15.78:55772) 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2accab4700 1 -- 172.21.15.78:0/3120173770 learned_addr learned my addr 172.21.15.78:0/3120173770 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.612+0000 7f2accab4700 1 -- 172.21.15.78:0/3120173770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac8108ec0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2accab4700 1 --2- 172.21.15.78:0/3120173770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac81080e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0098a0 tx=0x7f2abc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361fb49dde97ca7d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2ac77fe700 1 -- 172.21.15.78:0/3120173770 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2abc004160 con 0x7f2ac8107cc0 2024-03-16T10:56:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2ac77fe700 1 -- 172.21.15.78:0/3120173770 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2abc004300 con 0x7f2ac8107cc0 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2ac77fe700 1 -- 172.21.15.78:0/3120173770 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2abc011a80 con 0x7f2ac8107cc0 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2aced18700 1 -- 172.21.15.78:0/3120173770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 msgr2=0x7f2ac81080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2aced18700 1 --2- 172.21.15.78:0/3120173770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac81080e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0098a0 tx=0x7f2abc009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2aced18700 1 -- 172.21.15.78:0/3120173770 shutdown_connections 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2aced18700 1 --2- 172.21.15.78:0/3120173770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac81080e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2aced18700 1 -- 172.21.15.78:0/3120173770 >> 172.21.15.78:0/3120173770 conn(0x7f2ac81032c0 msgr2=0x7f2ac81056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2aced18700 1 -- 172.21.15.78:0/3120173770 shutdown_connections 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.613+0000 7f2aced18700 1 -- 172.21.15.78:0/3120173770 wait complete. 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.614+0000 7f2aced18700 1 Processor -- start 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.614+0000 7f2aced18700 1 -- start start 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.614+0000 7f2aced18700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac819bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.614+0000 7f2aced18700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac810c0f0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.614+0000 7f2accab4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac819bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.614+0000 7f2accab4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac819bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55778/0 (socket says 172.21.15.78:55778) 2024-03-16T10:56:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.614+0000 7f2accab4700 1 -- 172.21.15.78:0/194371697 learned_addr learned my addr 172.21.15.78:0/194371697 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.615+0000 7f2accab4700 1 -- 172.21.15.78:0/194371697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac819c0c0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.615+0000 7f2accab4700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac819bb80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2abc006af0 tx=0x7f2abc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.615+0000 7f2ac5ffb700 1 -- 172.21.15.78:0/194371697 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2abc003930 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.615+0000 7f2ac5ffb700 1 -- 172.21.15.78:0/194371697 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2abc003ad0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.615+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac819c350 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.615+0000 7f2ac5ffb700 1 -- 172.21.15.78:0/194371697 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2abc0195e0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.615+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac819f040 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.616+0000 7f2ac5ffb700 1 -- 172.21.15.78:0/194371697 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2abc0198e0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.616+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac8063900 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.616+0000 7f2ac5ffb700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f2ab8038ed0 0x7f2ab803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.616+0000 7f2ac5ffb700 1 -- 172.21.15.78:0/194371697 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2abc04d380 con 0x7f2ac8107cc0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.616+0000 7f2ac7fff700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f2ab8038ed0 0x7f2ab803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.617+0000 7f2ac7fff700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f2ab8038ed0 0x7f2ab803b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4006fd0 tx=0x7f2ab4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.621+0000 7f2ac5ffb700 1 -- 172.21.15.78:0/194371697 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2abc019cd0 con 0x7f2ac8107cc0 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.833+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f2ac8104550 con 0x7f2ab8038ed0 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.841+0000 7f2ac5ffb700 1 -- 172.21.15.78:0/194371697 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2ac8104550 con 0x7f2ab8038ed0 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f2ab8038ed0 msgr2=0x7f2ab803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f2ab8038ed0 0x7f2ab803b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4006fd0 tx=0x7f2ab4006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 msgr2=0x7f2ac819bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac819bb80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2abc006af0 tx=0x7f2abc011b70 comp rx=0 tx=0).stop 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 shutdown_connections 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f2ab8038ed0 0x7f2ab803b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 --2- 172.21.15.78:0/194371697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8107cc0 0x7f2ac819bb80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 >> 172.21.15.78:0/194371697 conn(0x7f2ac81032c0 msgr2=0x7f2ac8103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.843+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 shutdown_connections 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:47.844+0000 7f2aced18700 1 -- 172.21.15.78:0/194371697 wait complete. 2024-03-16T10:56:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mgr service... 2024-03-16T10:56:50.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:50 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:50.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:50 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.861+0000 7f3994b47700 1 Processor -- start 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.861+0000 7f3994b47700 1 -- start start 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f3994b47700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f3990107010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f3994b47700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f39901075e0 con 0x7f3990106bf0 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f398e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f3990107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f398e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f3990107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55786/0 (socket says 172.21.15.78:55786) 2024-03-16T10:56:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f398e59c700 1 -- 172.21.15.78:0/833722673 learned_addr learned my addr 172.21.15.78:0/833722673 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f398e59c700 1 -- 172.21.15.78:0/833722673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3990107df0 con 0x7f3990106bf0 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f398e59c700 1 --2- 172.21.15.78:0/833722673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f3990107010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39800098a0 tx=0x7f3980009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e1a9ba051687f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.862+0000 7f398d59a700 1 -- 172.21.15.78:0/833722673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3980004160 con 0x7f3990106bf0 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f398d59a700 1 -- 172.21.15.78:0/833722673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3980004300 con 0x7f3990106bf0 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f398d59a700 1 -- 172.21.15.78:0/833722673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3980011a50 con 0x7f3990106bf0 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f3994b47700 1 -- 172.21.15.78:0/833722673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 msgr2=0x7f3990107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f3994b47700 1 --2- 172.21.15.78:0/833722673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f3990107010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39800098a0 tx=0x7f3980009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f3994b47700 1 -- 172.21.15.78:0/833722673 shutdown_connections 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f3994b47700 1 --2- 172.21.15.78:0/833722673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f3990107010 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f3994b47700 1 -- 172.21.15.78:0/833722673 >> 172.21.15.78:0/833722673 conn(0x7f39901021b0 msgr2=0x7f39901045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f3994b47700 1 -- 172.21.15.78:0/833722673 shutdown_connections 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.863+0000 7f3994b47700 1 -- 172.21.15.78:0/833722673 wait complete. 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.864+0000 7f3994b47700 1 Processor -- start 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.864+0000 7f3994b47700 1 -- start start 2024-03-16T10:56:50.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.864+0000 7f3994b47700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f399019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.864+0000 7f3994b47700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f399010b1a0 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.864+0000 7f398e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f399019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.864+0000 7f398e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f399019aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55794/0 (socket says 172.21.15.78:55794) 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.864+0000 7f398e59c700 1 -- 172.21.15.78:0/805295251 learned_addr learned my addr 172.21.15.78:0/805295251 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.865+0000 7f398e59c700 1 -- 172.21.15.78:0/805295251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399019afe0 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.865+0000 7f398e59c700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f399019aaa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39800117a0 tx=0x7f3980011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.865+0000 7f397f7fe700 1 -- 172.21.15.78:0/805295251 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39800038b0 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.865+0000 7f397f7fe700 1 -- 172.21.15.78:0/805295251 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3980003a50 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.865+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399019b270 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.865+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f399019df10 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.865+0000 7f397f7fe700 1 -- 172.21.15.78:0/805295251 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3980019600 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.866+0000 7f397f7fe700 1 -- 172.21.15.78:0/805295251 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3980019900 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.866+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3990063900 con 0x7f3990106bf0 2024-03-16T10:56:50.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.866+0000 7f397f7fe700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f3978038f20 0x7f397803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.866+0000 7f397f7fe700 1 -- 172.21.15.78:0/805295251 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f398004d110 con 0x7f3990106bf0 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.866+0000 7f398dd9b700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f3978038f20 0x7f397803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.867+0000 7f398dd9b700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f3978038f20 0x7f397803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3984006fd0 tx=0x7f3984006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:49.871+0000 7f397f7fe700 1 -- 172.21.15.78:0/805295251 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f398001aa70 con 0x7f3990106bf0 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.092+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f39901033f0 con 0x7f3978038f20 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.099+0000 7f397f7fe700 1 -- 172.21.15.78:0/805295251 <== mgr.14120 v2:172.21.15.78:6800/3269664941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f39901033f0 con 0x7f3978038f20 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f3978038f20 msgr2=0x7f397803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f3978038f20 0x7f397803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3984006fd0 tx=0x7f3984006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 msgr2=0x7f399019aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f399019aaa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39800117a0 tx=0x7f3980011ad0 comp rx=0 tx=0).stop 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 shutdown_connections 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f3978038f20 0x7f397803b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 --2- 172.21.15.78:0/805295251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990106bf0 0x7f399019aaa0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 >> 172.21.15.78:0/805295251 conn(0x7f39901021b0 msgr2=0x7f3990102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:50.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 shutdown_connections 2024-03-16T10:56:50.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:50.102+0000 7f3994b47700 1 -- 172.21.15.78:0/805295251 wait complete. 2024-03-16T10:56:51.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:51 smithi078 ceph-mon[32235]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:56:51.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:51 smithi078 ceph-mon[32235]: Saving service mgr spec with placement count:2 2024-03-16T10:56:51.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:51 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f538af38700 1 Processor -- start 2024-03-16T10:56:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f538af38700 1 -- start start 2024-03-16T10:56:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f538af38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f5384108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f538af38700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f53841086f0 con 0x7f5384107d00 2024-03-16T10:56:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f5388cd4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f5384108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f5388cd4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f5384108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55810/0 (socket says 172.21.15.78:55810) 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f5388cd4700 1 -- 172.21.15.78:0/4034414137 learned_addr learned my addr 172.21.15.78:0/4034414137 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f5388cd4700 1 -- 172.21.15.78:0/4034414137 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5384108f00 con 0x7f5384107d00 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.111+0000 7f5388cd4700 1 --2- 172.21.15.78:0/4034414137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f5384108120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f53780098a0 tx=0x7f5378009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b3c01b0f440e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f53837fe700 1 -- 172.21.15.78:0/4034414137 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5378004160 con 0x7f5384107d00 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f53837fe700 1 -- 172.21.15.78:0/4034414137 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5378004300 con 0x7f5384107d00 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f53837fe700 1 -- 172.21.15.78:0/4034414137 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5378011a50 con 0x7f5384107d00 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f538af38700 1 -- 172.21.15.78:0/4034414137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 msgr2=0x7f5384108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f538af38700 1 --2- 172.21.15.78:0/4034414137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f5384108120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f53780098a0 tx=0x7f5378009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f538af38700 1 -- 172.21.15.78:0/4034414137 shutdown_connections 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f538af38700 1 --2- 172.21.15.78:0/4034414137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f5384108120 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f538af38700 1 -- 172.21.15.78:0/4034414137 >> 172.21.15.78:0/4034414137 conn(0x7f5384103340 msgr2=0x7f5384105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f538af38700 1 -- 172.21.15.78:0/4034414137 shutdown_connections 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.112+0000 7f538af38700 1 -- 172.21.15.78:0/4034414137 wait complete. 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f538af38700 1 Processor -- start 2024-03-16T10:56:52.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f538af38700 1 -- start start 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f538af38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f538407daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f538af38700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f538410c2b0 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f5388cd4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f538407daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f5388cd4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f538407daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55824/0 (socket says 172.21.15.78:55824) 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f5388cd4700 1 -- 172.21.15.78:0/1742123234 learned_addr learned my addr 172.21.15.78:0/1742123234 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f5388cd4700 1 -- 172.21.15.78:0/1742123234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538407e030 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.113+0000 7f5388cd4700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f538407daf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5378011670 tx=0x7f5378011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.114+0000 7f5381ffb700 1 -- 172.21.15.78:0/1742123234 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53780038b0 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.114+0000 7f5381ffb700 1 -- 172.21.15.78:0/1742123234 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5378003a50 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.114+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538407c220 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.114+0000 7f5381ffb700 1 -- 172.21.15.78:0/1742123234 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53780195c0 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.114+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538407c760 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.115+0000 7f5381ffb700 1 -- 172.21.15.78:0/1742123234 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5378019890 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.115+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5384063900 con 0x7f5384107d00 2024-03-16T10:56:52.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.115+0000 7f5381ffb700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5374038f20 0x7f537403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.115+0000 7f5381ffb700 1 -- 172.21.15.78:0/1742123234 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f537804d360 con 0x7f5384107d00 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.115+0000 7f5383fff700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5374038f20 0x7f537403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.115+0000 7f5383fff700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5374038f20 0x7f537403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5370006fd0 tx=0x7f5370006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.120+0000 7f5381ffb700 1 -- 172.21.15.78:0/1742123234 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f537801aa70 con 0x7f5384107d00 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.334+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f538407d000 con 0x7f5384107d00 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.337+0000 7f5381ffb700 1 -- 172.21.15.78:0/1742123234 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f537802d960 con 0x7f5384107d00 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5374038f20 msgr2=0x7f537403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5374038f20 0x7f537403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5370006fd0 tx=0x7f5370006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 msgr2=0x7f538407daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f538407daf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5378011670 tx=0x7f5378011ab0 comp rx=0 tx=0).stop 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 shutdown_connections 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5374038f20 0x7f537403b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 --2- 172.21.15.78:0/1742123234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5384107d00 0x7f538407daf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 >> 172.21.15.78:0/1742123234 conn(0x7f5384103340 msgr2=0x7f5384103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 shutdown_connections 2024-03-16T10:56:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:51.339+0000 7f538af38700 1 -- 172.21.15.78:0/1742123234 wait complete. 2024-03-16T10:56:52.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:52 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1742123234' entity='client.admin' 2024-03-16T10:56:52.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:52 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:53.759 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.948+0000 7f10c1c11700 1 Processor -- start 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.948+0000 7f10c1c11700 1 -- start start 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.948+0000 7f10c1c11700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc10ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.948+0000 7f10c1c11700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc07be30 con 0x7f10bc108720 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.948+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc10ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.948+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc10ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55826/0 (socket says 172.21.15.78:55826) 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.948+0000 7f10bb7fe700 1 -- 172.21.15.78:0/858952961 learned_addr learned my addr 172.21.15.78:0/858952961 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.949+0000 7f10bb7fe700 1 -- 172.21.15.78:0/858952961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc10b050 con 0x7f10bc108720 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.949+0000 7f10bb7fe700 1 --2- 172.21.15.78:0/858952961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc10ab10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0098a0 tx=0x7f10ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48223d03df1fbe19 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.949+0000 7f10ba7fc700 1 -- 172.21.15.78:0/858952961 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac004160 con 0x7f10bc108720 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.949+0000 7f10ba7fc700 1 -- 172.21.15.78:0/858952961 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10ac004300 con 0x7f10bc108720 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.949+0000 7f10ba7fc700 1 -- 172.21.15.78:0/858952961 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac011a50 con 0x7f10bc108720 2024-03-16T10:56:53.760 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 -- 172.21.15.78:0/858952961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 msgr2=0x7f10bc10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 --2- 172.21.15.78:0/858952961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc10ab10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0098a0 tx=0x7f10ac009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 -- 172.21.15.78:0/858952961 shutdown_connections 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 --2- 172.21.15.78:0/858952961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc10ab10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 -- 172.21.15.78:0/858952961 >> 172.21.15.78:0/858952961 conn(0x7f10bc1021d0 msgr2=0x7f10bc1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 -- 172.21.15.78:0/858952961 shutdown_connections 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 -- 172.21.15.78:0/858952961 wait complete. 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.950+0000 7f10c1c11700 1 Processor -- start 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10c1c11700 1 -- start start 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10c1c11700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10c1c11700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc10c060 con 0x7f10bc108720 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55830/0 (socket says 172.21.15.78:55830) 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10bb7fe700 1 -- 172.21.15.78:0/1197319974 learned_addr learned my addr 172.21.15.78:0/1197319974 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:53.761 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10bb7fe700 1 -- 172.21.15.78:0/1197319974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc19bfe0 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.951+0000 7f10bb7fe700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc19baa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10ac011a00 tx=0x7f10ac011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.952+0000 7f10b8ff9700 1 -- 172.21.15.78:0/1197319974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac003af0 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.952+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10bc19c270 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.952+0000 7f10b8ff9700 1 -- 172.21.15.78:0/1197319974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10ac01a8e0 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.952+0000 7f10b8ff9700 1 -- 172.21.15.78:0/1197319974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac024840 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.952+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10bc19ef60 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.953+0000 7f10b8ff9700 1 -- 172.21.15.78:0/1197319974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f10ac022020 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.953+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10bc063900 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.953+0000 7f10b8ff9700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f10a4038f70 0x7f10a403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.953+0000 7f10b8ff9700 1 -- 172.21.15.78:0/1197319974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f10ac04d250 con 0x7f10bc108720 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.953+0000 7f10baffd700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f10a4038f70 0x7f10a403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.953+0000 7f10baffd700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f10a4038f70 0x7f10a403b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10b0006fd0 tx=0x7f10b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:53.762 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:52.958+0000 7f10b8ff9700 1 -- 172.21.15.78:0/1197319974 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10ac019c70 con 0x7f10bc108720 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.164+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f10bc19f580 con 0x7f10bc108720 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.168+0000 7f10b8ff9700 1 -- 172.21.15.78:0/1197319974 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f10ac024e20 con 0x7f10bc108720 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f10a4038f70 msgr2=0x7f10a403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f10a4038f70 0x7f10a403b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10b0006fd0 tx=0x7f10b0006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 msgr2=0x7f10bc19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc19baa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10ac011a00 tx=0x7f10ac011ae0 comp rx=0 tx=0).stop 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 shutdown_connections 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f10a4038f70 0x7f10a403b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 --2- 172.21.15.78:0/1197319974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10bc108720 0x7f10bc19baa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 >> 172.21.15.78:0/1197319974 conn(0x7f10bc1021d0 msgr2=0x7f10bc102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 shutdown_connections 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:53.170+0000 7f10c1c11700 1 -- 172.21.15.78:0/1197319974 wait complete. 2024-03-16T10:56:53.763 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling the dashboard module... 2024-03-16T10:56:54.146 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:54 smithi078 ceph-mon[32235]: from='mgr.14120 172.21.15.78:0/594798726' entity='mgr.x' 2024-03-16T10:56:54.147 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:54 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1197319974' entity='client.admin' 2024-03-16T10:56:55.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:55 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4273096991' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-16T10:56:55.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: ignoring --setuser ceph since I am not root 2024-03-16T10:56:55.310 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: ignoring --setgroup ceph since I am not root 2024-03-16T10:56:55.881 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:55.600+0000 7f2df879e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:56:55.881 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:55.694+0000 7f2df879e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.165+0000 7f7b68689700 1 Processor -- start 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.166+0000 7f7b68689700 1 -- start start 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.166+0000 7f7b68689700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b601063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.166+0000 7f7b68689700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60106990 con 0x7f7b60105fa0 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.166+0000 7f7b66425700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b601063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.166+0000 7f7b66425700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b601063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55846/0 (socket says 172.21.15.78:55846) 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.166+0000 7f7b66425700 1 -- 172.21.15.78:0/4168566947 learned_addr learned my addr 172.21.15.78:0/4168566947 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.166+0000 7f7b66425700 1 -- 172.21.15.78:0/4168566947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b60108fd0 con 0x7f7b60105fa0 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.167+0000 7f7b66425700 1 --2- 172.21.15.78:0/4168566947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b601063c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0098a0 tx=0x7f7b5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f35b0fc7f2415e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.167+0000 7f7b65423700 1 -- 172.21.15.78:0/4168566947 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b5c004160 con 0x7f7b60105fa0 2024-03-16T10:56:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.167+0000 7f7b65423700 1 -- 172.21.15.78:0/4168566947 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b5c004300 con 0x7f7b60105fa0 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.167+0000 7f7b65423700 1 -- 172.21.15.78:0/4168566947 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b5c011a50 con 0x7f7b60105fa0 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.167+0000 7f7b68689700 1 -- 172.21.15.78:0/4168566947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 msgr2=0x7f7b601063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.167+0000 7f7b68689700 1 --2- 172.21.15.78:0/4168566947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b601063c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0098a0 tx=0x7f7b5c009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 -- 172.21.15.78:0/4168566947 shutdown_connections 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 --2- 172.21.15.78:0/4168566947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b601063c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 -- 172.21.15.78:0/4168566947 >> 172.21.15.78:0/4168566947 conn(0x7f7b601018f0 msgr2=0x7f7b60103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 -- 172.21.15.78:0/4168566947 shutdown_connections 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 -- 172.21.15.78:0/4168566947 wait complete. 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 Processor -- start 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 -- start start 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b60108850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.168+0000 7f7b68689700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6010c200 con 0x7f7b60105fa0 2024-03-16T10:56:55.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b66425700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b60108850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b66425700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b60108850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55858/0 (socket says 172.21.15.78:55858) 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b66425700 1 -- 172.21.15.78:0/4273096991 learned_addr learned my addr 172.21.15.78:0/4273096991 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b66425700 1 -- 172.21.15.78:0/4273096991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b60108d90 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b66425700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b60108850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0117a0 tx=0x7f7b5c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b5c0038b0 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b5c003a50 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b60106f80 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b5c019600 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.169+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b601074c0 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.170+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f7b5c019900 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.170+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b6010c380 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.171+0000 7f7b577fe700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f7b4c038f20 0x7f7b4c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.171+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7b5c04d070 con 0x7f7b60105fa0 2024-03-16T10:56:55.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.171+0000 7f7b65c24700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f7b4c038f20 0x7f7b4c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.171+0000 7f7b65c24700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f7b4c038f20 0x7f7b4c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7b50006fd0 tx=0x7f7b50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.178+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b5c04ca00 con 0x7f7b60105fa0 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:54.439+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f7b60107c20 con 0x7f7b60105fa0 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.036+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f7b5c0234e0 con 0x7f7b60105fa0 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.037+0000 7f7b577fe700 1 -- 172.21.15.78:0/4273096991 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f7b5c01aa70 con 0x7f7b60105fa0 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f7b4c038f20 msgr2=0x7f7b4c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f7b4c038f20 0x7f7b4c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7b50006fd0 tx=0x7f7b50006e40 comp rx=0 tx=0).stop 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 msgr2=0x7f7b60108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b60108850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0117a0 tx=0x7f7b5c011ad0 comp rx=0 tx=0).stop 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 shutdown_connections 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f7b4c038f20 0x7f7b4c03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 --2- 172.21.15.78:0/4273096991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b60105fa0 0x7f7b60108850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:55.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 >> 172.21.15.78:0/4273096991 conn(0x7f7b601018f0 msgr2=0x7f7b60102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:55.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 shutdown_connections 2024-03-16T10:56:55.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:55.039+0000 7f7b68689700 1 -- 172.21.15.78:0/4273096991 wait complete. 2024-03-16T10:56:56.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:56 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4273096991' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-16T10:56:56.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:56 smithi078 ceph-mon[32235]: mgrmap e9: x(active, since 20s) 2024-03-16T10:56:56.449 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:56.351+0000 7f2df879e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:56:56.449 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:56.448+0000 7f2df879e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:56:56.776 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:56.593+0000 7f2df879e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:56:56.777 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:56.692+0000 7f2df879e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:56:56.777 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:56.775+0000 7f2df879e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.348+0000 7f5a98a68700 1 Processor -- start 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.349+0000 7f5a98a68700 1 -- start start 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.349+0000 7f5a98a68700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a94106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:57.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.349+0000 7f5a98a68700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a94063550 con 0x7f5a941065f0 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.349+0000 7f5a9259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a94106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.349+0000 7f5a9259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a94106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45284/0 (socket says 172.21.15.78:45284) 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.349+0000 7f5a9259c700 1 -- 172.21.15.78:0/847402231 learned_addr learned my addr 172.21.15.78:0/847402231 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.350+0000 7f5a9259c700 1 -- 172.21.15.78:0/847402231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a94106f50 con 0x7f5a941065f0 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.350+0000 7f5a9259c700 1 --2- 172.21.15.78:0/847402231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a94106a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5a840098a0 tx=0x7f5a84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720d35ee3a6c5fad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.350+0000 7f5a9159a700 1 -- 172.21.15.78:0/847402231 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a84004160 con 0x7f5a941065f0 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.350+0000 7f5a9159a700 1 -- 172.21.15.78:0/847402231 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a84004300 con 0x7f5a941065f0 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.350+0000 7f5a9159a700 1 -- 172.21.15.78:0/847402231 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a84011a50 con 0x7f5a941065f0 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.350+0000 7f5a98a68700 1 -- 172.21.15.78:0/847402231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 msgr2=0x7f5a94106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.350+0000 7f5a98a68700 1 --2- 172.21.15.78:0/847402231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a94106a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5a840098a0 tx=0x7f5a84009bb0 comp rx=0 tx=0).stop 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.351+0000 7f5a98a68700 1 -- 172.21.15.78:0/847402231 shutdown_connections 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.351+0000 7f5a98a68700 1 --2- 172.21.15.78:0/847402231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a94106a10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.351+0000 7f5a98a68700 1 -- 172.21.15.78:0/847402231 >> 172.21.15.78:0/847402231 conn(0x7f5a941021b0 msgr2=0x7f5a941045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.351+0000 7f5a98a68700 1 -- 172.21.15.78:0/847402231 shutdown_connections 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.351+0000 7f5a98a68700 1 -- 172.21.15.78:0/847402231 wait complete. 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.351+0000 7f5a98a68700 1 Processor -- start 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.351+0000 7f5a98a68700 1 -- start start 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a98a68700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a9419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a98a68700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a94107fc0 con 0x7f5a941065f0 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a9259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a9419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a9259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a9419dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45288/0 (socket says 172.21.15.78:45288) 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a9259c700 1 -- 172.21.15.78:0/837895382 learned_addr learned my addr 172.21.15.78:0/837895382 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a9259c700 1 -- 172.21.15.78:0/837895382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9419e290 con 0x7f5a941065f0 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a9259c700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a9419dd50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a84011a00 tx=0x7f5a84011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a837fe700 1 -- 172.21.15.78:0/837895382 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a840039e0 con 0x7f5a941065f0 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a837fe700 1 -- 172.21.15.78:0/837895382 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a8401a8e0 con 0x7f5a941065f0 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a98a68700 1 -- 172.21.15.78:0/837895382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9419e520 con 0x7f5a941065f0 2024-03-16T10:56:57.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a837fe700 1 -- 172.21.15.78:0/837895382 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a84024760 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.352+0000 7f5a98a68700 1 -- 172.21.15.78:0/837895382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a941a1210 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.353+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a78005330 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.354+0000 7f5a837fe700 1 -- 172.21.15.78:0/837895382 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f5a84022020 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.354+0000 7f5a837fe700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 0x7f5a7c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.354+0000 7f5a837fe700 1 -- 172.21.15.78:0/837895382 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5a8404d690 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.354+0000 7f5a91d9b700 1 -- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 msgr2=0x7f5a7c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3269664941 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.354+0000 7f5a91d9b700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 0x7f5a7c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.358+0000 7f5a837fe700 1 -- 172.21.15.78:0/837895382 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a8404c0c0 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.555+0000 7f5a91d9b700 1 -- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 msgr2=0x7f5a7c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3269664941 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.555+0000 7f5a91d9b700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 0x7f5a7c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.616+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f5a780058f0 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.617+0000 7f5a837fe700 1 -- 172.21.15.78:0/837895382 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f5a8404c440 con 0x7f5a941065f0 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 msgr2=0x7f5a7c03b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:56:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 0x7f5a7c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 msgr2=0x7f5a9419dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a9419dd50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a84011a00 tx=0x7f5a84011ae0 comp rx=0 tx=0).stop 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 shutdown_connections 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f5a7c038fc0 0x7f5a7c03b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 --2- 172.21.15.78:0/837895382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a941065f0 0x7f5a9419dd50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 >> 172.21.15.78:0/837895382 conn(0x7f5a941021b0 msgr2=0x7f5a94194a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 shutdown_connections 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:56.619+0000 7f5a817fa700 1 -- 172.21.15.78:0/837895382 wait complete. 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-03-16T10:56:57.159 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 9... 2024-03-16T10:56:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:56:57 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/837895382' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T10:56:57.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:57 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:57.041+0000 7f2df879e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:56:57.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:57 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:57.151+0000 7f2df879e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:56:57.310 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:57 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:57.246+0000 7f2df879e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:56:57.770 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:57 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:57.342+0000 7f2df879e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:56:57.770 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:57 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:57.523+0000 7f2df879e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:56:58.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:58 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:58.475+0000 7f2df879e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:56:58.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:58 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:58.605+0000 7f2df879e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:56:59.224 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:58 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:58.936+0000 7f2df879e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:56:59.224 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:59 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:59.026+0000 7f2df879e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:56:59.224 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:59 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:59.223+0000 7f2df879e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:56:59.559 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:59 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:59.408+0000 7f2df879e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:56:59.953 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:59 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:59.694+0000 7f2df879e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:56:59.953 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:56:59 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:56:59.952+0000 7f2df879e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:57:00.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:57:00 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:57:00.190+0000 7f2df879e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:57:01.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:57:00 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:57:00.989+0000 7f2df879e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:57:01.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:57:01.555+0000 7f2df879e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:57:01.809 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:57:01.653+0000 7f2df879e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:57:02.309 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T10:57:01.870+0000 7f2df879e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:57:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: Active manager daemon x restarted 2024-03-16T10:57:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: Activating manager daemon x 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: osdmap e3: 0 total, 0 up, 0 in 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: mgrmap e10: x(active, starting, since 0.00316677s) 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: Manager daemon x is now available 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:03.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T10:57:03.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T10:57:03.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:02 smithi078 ceph-mon[32235]: mgrmap e11: x(active, since 1.00588s) 2024-03-16T10:57:03.478 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f76047ed700 1 Processor -- start 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f76047ed700 1 -- start start 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f76047ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc106990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f76047ed700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc07bdf0 con 0x7f75fc106570 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f7602589700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc106990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f7602589700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc106990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45294/0 (socket says 172.21.15.78:45294) 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f7602589700 1 -- 172.21.15.78:0/2104270667 learned_addr learned my addr 172.21.15.78:0/2104270667 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.564+0000 7f7602589700 1 -- 172.21.15.78:0/2104270667 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc106ed0 con 0x7f75fc106570 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f7602589700 1 --2- 172.21.15.78:0/2104270667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc106990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75f80098a0 tx=0x7f75f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6052e47fc2e4f667 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f7601587700 1 -- 172.21.15.78:0/2104270667 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f8004160 con 0x7f75fc106570 2024-03-16T10:57:03.479 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f7601587700 1 -- 172.21.15.78:0/2104270667 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75f8004300 con 0x7f75fc106570 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f7601587700 1 -- 172.21.15.78:0/2104270667 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f8011a50 con 0x7f75fc106570 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f76047ed700 1 -- 172.21.15.78:0/2104270667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 msgr2=0x7f75fc106990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f76047ed700 1 --2- 172.21.15.78:0/2104270667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc106990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75f80098a0 tx=0x7f75f8009bb0 comp rx=0 tx=0).stop 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f76047ed700 1 -- 172.21.15.78:0/2104270667 shutdown_connections 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f76047ed700 1 --2- 172.21.15.78:0/2104270667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc106990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.565+0000 7f76047ed700 1 -- 172.21.15.78:0/2104270667 >> 172.21.15.78:0/2104270667 conn(0x7f75fc1021d0 msgr2=0x7f75fc1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f76047ed700 1 -- 172.21.15.78:0/2104270667 shutdown_connections 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f76047ed700 1 -- 172.21.15.78:0/2104270667 wait complete. 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f76047ed700 1 Processor -- start 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f76047ed700 1 -- start start 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f76047ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f76047ed700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc107f40 con 0x7f75fc106570 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f7602589700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:03.480 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f7602589700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc19dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45310/0 (socket says 172.21.15.78:45310) 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.566+0000 7f7602589700 1 -- 172.21.15.78:0/1601202383 learned_addr learned my addr 172.21.15.78:0/1601202383 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.567+0000 7f7602589700 1 -- 172.21.15.78:0/1601202383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc19e210 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.567+0000 7f7602589700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc19dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f75f8011a20 tx=0x7f75f8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.567+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f80039c0 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.567+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75fc19e4a0 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.567+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75f8003b60 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.567+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f801a6b0 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.567+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75fc1a1180 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.568+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f75f8024430 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.568+0000 7f75f37fe700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.568+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 --> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f75e803baf0 con 0x7f75e8038f20 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.568+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f75f804d4f0 con 0x7f75fc106570 2024-03-16T10:57:03.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.568+0000 7f7601d88700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 msgr2=0x7f75e803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3269664941 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.568+0000 7f7601d88700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.769+0000 7f7601d88700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 msgr2=0x7f75e803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3269664941 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:57.769+0000 7f7601d88700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:58.170+0000 7f7601d88700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 msgr2=0x7f75e803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3269664941 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:58.170+0000 7f7601d88700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:58.971+0000 7f7601d88700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 msgr2=0x7f75e803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3269664941 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:56:58.971+0000 7f7601d88700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:00.572+0000 7f7601d88700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 msgr2=0x7f75e803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3269664941 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:00.572+0000 7f7601d88700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:01.885+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f75f804bd10 con 0x7f75fc106570 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:01.885+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 msgr2=0x7f75e803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:01.885+0000 7f75f37fe700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.886+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f75f8029cf0 con 0x7f75fc106570 2024-03-16T10:57:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.886+0000 7f75f37fe700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75e803c9a0 0x7f75e803ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.886+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f75f801cec0 con 0x7f75e803c9a0 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.887+0000 7f7601d88700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75e803c9a0 0x7f75e803ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.887+0000 7f7601d88700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75e803c9a0 0x7f75e803ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75ec003a10 tx=0x7f75ec0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.888+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f75f801cec0 con 0x7f75e803c9a0 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f75fc063900 con 0x7f75e803c9a0 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f75f37fe700 1 -- 172.21.15.78:0/1601202383 <== mgr.14152 v2:172.21.15.78:6800/4254906951 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f75fc063900 con 0x7f75e803c9a0 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75e803c9a0 msgr2=0x7f75e803ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75e803c9a0 0x7f75e803ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75ec003a10 tx=0x7f75ec0092b0 comp rx=0 tx=0).stop 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 msgr2=0x7f75fc19dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc19dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f75f8011a20 tx=0x7f75f8011b00 comp rx=0 tx=0).stop 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 shutdown_connections 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75e803c9a0 0x7f75e803ed90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75fc106570 0x7f75fc19dcd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 --2- 172.21.15.78:0/1601202383 >> [v2:172.21.15.78:6800/3269664941,v1:172.21.15.78:6801/3269664941] conn(0x7f75e8038f20 0x7f75e803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.901+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 >> 172.21.15.78:0/1601202383 conn(0x7f75fc1021d0 msgr2=0x7f75fc194990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.902+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 shutdown_connections 2024-03-16T10:57:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:02.902+0000 7f76047ed700 1 -- 172.21.15.78:0/1601202383 wait complete. 2024-03-16T10:57:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 9 is available 2024-03-16T10:57:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:Generating a dashboard self-signed certificate... 2024-03-16T10:57:04.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:03 smithi078 ceph-mon[32235]: [16/Mar/2024:10:57:02] ENGINE Bus STARTING 2024-03-16T10:57:04.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:03 smithi078 ceph-mon[32235]: [16/Mar/2024:10:57:02] ENGINE Serving on http://172.21.15.78:8765 2024-03-16T10:57:04.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:03 smithi078 ceph-mon[32235]: [16/Mar/2024:10:57:02] ENGINE Serving on https://172.21.15.78:7150 2024-03-16T10:57:04.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:03 smithi078 ceph-mon[32235]: [16/Mar/2024:10:57:02] ENGINE Bus STARTED 2024-03-16T10:57:04.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:03 smithi078 ceph-mon[32235]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T10:57:04.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:03 smithi078 ceph-mon[32235]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T10:57:05.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:04 smithi078 ceph-mon[32235]: mgrmap e12: x(active, since 2s) 2024-03-16T10:57:05.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:05.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:05.153 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.146+0000 7fcf13541700 1 Processor -- start 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.146+0000 7fcf13541700 1 -- start start 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.146+0000 7fcf13541700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c1063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.146+0000 7fcf13541700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c106990 con 0x7fcf0c105fa0 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf112dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c1063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf112dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c1063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45376/0 (socket says 172.21.15.78:45376) 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf112dd700 1 -- 172.21.15.78:0/2369403653 learned_addr learned my addr 172.21.15.78:0/2369403653 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf112dd700 1 -- 172.21.15.78:0/2369403653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c108f30 con 0x7fcf0c105fa0 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf112dd700 1 --2- 172.21.15.78:0/2369403653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c1063c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0098a0 tx=0x7fcefc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68ddcfd3d1de8f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf03fff700 1 -- 172.21.15.78:0/2369403653 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcefc004160 con 0x7fcf0c105fa0 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf03fff700 1 -- 172.21.15.78:0/2369403653 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcefc004300 con 0x7fcf0c105fa0 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.147+0000 7fcf03fff700 1 -- 172.21.15.78:0/2369403653 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcefc011a50 con 0x7fcf0c105fa0 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 -- 172.21.15.78:0/2369403653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 msgr2=0x7fcf0c1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 --2- 172.21.15.78:0/2369403653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c1063c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0098a0 tx=0x7fcefc009bb0 comp rx=0 tx=0).stop 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 -- 172.21.15.78:0/2369403653 shutdown_connections 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 --2- 172.21.15.78:0/2369403653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c1063c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 -- 172.21.15.78:0/2369403653 >> 172.21.15.78:0/2369403653 conn(0x7fcf0c101930 msgr2=0x7fcf0c103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 -- 172.21.15.78:0/2369403653 shutdown_connections 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 -- 172.21.15.78:0/2369403653 wait complete. 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.148+0000 7fcf13541700 1 Processor -- start 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf13541700 1 -- start start 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf13541700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c19bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:05.161 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf13541700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c10c2e0 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf112dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c19bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf112dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c19bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45388/0 (socket says 172.21.15.78:45388) 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf112dd700 1 -- 172.21.15.78:0/121717658 learned_addr learned my addr 172.21.15.78:0/121717658 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf112dd700 1 -- 172.21.15.78:0/121717658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c19c0a0 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf112dd700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c19bb60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0117a0 tx=0x7fcefc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf027fc700 1 -- 172.21.15.78:0/121717658 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcefc0038b0 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf027fc700 1 -- 172.21.15.78:0/121717658 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcefc003a50 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0c19c330 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf027fc700 1 -- 172.21.15.78:0/121717658 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcefc019600 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.149+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0c19f020 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.150+0000 7fcf027fc700 1 -- 172.21.15.78:0/121717658 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fcefc0198b0 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.150+0000 7fcf027fc700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcef8038fd0 0x7fcef803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.150+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf0c063900 con 0x7fcf0c105fa0 2024-03-16T10:57:05.162 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.151+0000 7fcf027fc700 1 -- 172.21.15.78:0/121717658 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcefc04d370 con 0x7fcf0c105fa0 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.151+0000 7fcf10adc700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcef8038fd0 0x7fcef803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.151+0000 7fcf10adc700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcef8038fd0 0x7fcef803b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf08006fd0 tx=0x7fcf08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.156+0000 7fcf027fc700 1 -- 172.21.15.78:0/121717658 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcefc01a870 con 0x7fcf0c105fa0 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.366+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fcf0c102b50 con 0x7fcef8038fd0 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.567+0000 7fcf027fc700 1 -- 172.21.15.78:0/121717658 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fcf0c102b50 con 0x7fcef8038fd0 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcef8038fd0 msgr2=0x7fcef803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcef8038fd0 0x7fcef803b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf08006fd0 tx=0x7fcf08006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 msgr2=0x7fcf0c19bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c19bb60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0117a0 tx=0x7fcefc011ad0 comp rx=0 tx=0).stop 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 shutdown_connections 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcef8038fd0 0x7fcef803b490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 --2- 172.21.15.78:0/121717658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c105fa0 0x7fcf0c19bb60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 >> 172.21.15.78:0/121717658 conn(0x7fcf0c101930 msgr2=0x7fcf0c102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 shutdown_connections 2024-03-16T10:57:05.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:04.569+0000 7fcf13541700 1 -- 172.21.15.78:0/121717658 wait complete. 2024-03-16T10:57:05.164 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial admin user... 2024-03-16T10:57:06.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:05 smithi078 ceph-mon[32235]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:06.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:05 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ji68KTog8fMDU9p/qqWRf.fLc4MngXfZoejOR0ovxdV5T3SdkB8bO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710586626, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.589+0000 7f27b596c700 1 Processor -- start 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.589+0000 7f27b596c700 1 -- start start 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27b596c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b010ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27b596c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f27b007be30 con 0x7f27b0108730 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27aeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b010ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27aeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b010ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38458/0 (socket says 172.21.15.78:38458) 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27aeffd700 1 -- 172.21.15.78:0/1601538612 learned_addr learned my addr 172.21.15.78:0/1601538612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:06.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27aeffd700 1 -- 172.21.15.78:0/1601538612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b010b060 con 0x7f27b0108730 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27aeffd700 1 --2- 172.21.15.78:0/1601538612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b010ab20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f27a00098a0 tx=0x7f27a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e50413f14cd39f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27adffb700 1 -- 172.21.15.78:0/1601538612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27a0004160 con 0x7f27b0108730 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27adffb700 1 -- 172.21.15.78:0/1601538612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27a0004300 con 0x7f27b0108730 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.590+0000 7f27adffb700 1 -- 172.21.15.78:0/1601538612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27a0011a50 con 0x7f27b0108730 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.591+0000 7f27b596c700 1 -- 172.21.15.78:0/1601538612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 msgr2=0x7f27b010ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.591+0000 7f27b596c700 1 --2- 172.21.15.78:0/1601538612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b010ab20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f27a00098a0 tx=0x7f27a0009bb0 comp rx=0 tx=0).stop 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.591+0000 7f27b596c700 1 -- 172.21.15.78:0/1601538612 shutdown_connections 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.591+0000 7f27b596c700 1 --2- 172.21.15.78:0/1601538612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b010ab20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.591+0000 7f27b596c700 1 -- 172.21.15.78:0/1601538612 >> 172.21.15.78:0/1601538612 conn(0x7f27b0102180 msgr2=0x7f27b01045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.591+0000 7f27b596c700 1 -- 172.21.15.78:0/1601538612 shutdown_connections 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.591+0000 7f27b596c700 1 -- 172.21.15.78:0/1601538612 wait complete. 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.592+0000 7f27b596c700 1 Processor -- start 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.592+0000 7f27b596c700 1 -- start start 2024-03-16T10:57:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27b596c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b019bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27b596c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f27b010c070 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27aeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b019bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27aeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b019bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38462/0 (socket says 172.21.15.78:38462) 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27aeffd700 1 -- 172.21.15.78:0/2946332726 learned_addr learned my addr 172.21.15.78:0/2946332726 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27aeffd700 1 -- 172.21.15.78:0/2946332726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b019c050 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27aeffd700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b019bb10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27a0011670 tx=0x7f27a0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27b496a700 1 -- 172.21.15.78:0/2946332726 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27a00038b0 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27b496a700 1 -- 172.21.15.78:0/2946332726 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27a0003a50 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b019c2e0 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27b496a700 1 -- 172.21.15.78:0/2946332726 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27a00195c0 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.593+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b019efd0 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.594+0000 7f27b496a700 1 -- 172.21.15.78:0/2946332726 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f27a00198e0 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.594+0000 7f27b496a700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2798038f20 0x7f279803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.594+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b0063900 con 0x7f27b0108730 2024-03-16T10:57:06.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.594+0000 7f27b496a700 1 -- 172.21.15.78:0/2946332726 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f27a004d580 con 0x7f27b0108730 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.595+0000 7f27ae7fc700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2798038f20 0x7f279803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.595+0000 7f27ae7fc700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2798038f20 0x7f279803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006fd0 tx=0x7f27a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.600+0000 7f27b496a700 1 -- 172.21.15.78:0/2946332726 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27a001aa70 con 0x7f27b0108730 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:05.828+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f27b019f260 con 0x7f2798038f20 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.056+0000 7f27b496a700 1 -- 172.21.15.78:0/2946332726 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f27b019f260 con 0x7f2798038f20 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2798038f20 msgr2=0x7f279803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2798038f20 0x7f279803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006fd0 tx=0x7f27a4006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 msgr2=0x7f27b019bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b019bb10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27a0011670 tx=0x7f27a0011ab0 comp rx=0 tx=0).stop 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 shutdown_connections 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2798038f20 0x7f279803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 --2- 172.21.15.78:0/2946332726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27b0108730 0x7f27b019bb10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 >> 172.21.15.78:0/2946332726 conn(0x7f27b0102180 msgr2=0x7f27b0102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:06.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 shutdown_connections 2024-03-16T10:57:06.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:06.058+0000 7f27b596c700 1 -- 172.21.15.78:0/2946332726 wait complete. 2024-03-16T10:57:06.653 INFO:teuthology.orchestra.run.smithi078.stdout:Fetching dashboard port number... 2024-03-16T10:57:07.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:07 smithi078 ceph-mon[32235]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:07.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:07 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:08.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 8443 2024-03-16T10:57:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.094+0000 7f845dd9e700 1 Processor -- start 2024-03-16T10:57:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f845dd9e700 1 -- start start 2024-03-16T10:57:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f845dd9e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f845dd9e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f84580635b0 con 0x7f84581087a0 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f84577fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f84577fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38466/0 (socket says 172.21.15.78:38466) 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f84577fe700 1 -- 172.21.15.78:0/2573708406 learned_addr learned my addr 172.21.15.78:0/2573708406 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f84577fe700 1 -- 172.21.15.78:0/2573708406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845810b0d0 con 0x7f84581087a0 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.095+0000 7f84577fe700 1 --2- 172.21.15.78:0/2573708406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845810ab90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f84480098a0 tx=0x7f8448009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa31aacf9cc64a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f84567fc700 1 -- 172.21.15.78:0/2573708406 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8448004160 con 0x7f84581087a0 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f84567fc700 1 -- 172.21.15.78:0/2573708406 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8448004300 con 0x7f84581087a0 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f84567fc700 1 -- 172.21.15.78:0/2573708406 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8448011a50 con 0x7f84581087a0 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f845dd9e700 1 -- 172.21.15.78:0/2573708406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 msgr2=0x7f845810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f845dd9e700 1 --2- 172.21.15.78:0/2573708406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845810ab90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f84480098a0 tx=0x7f8448009bb0 comp rx=0 tx=0).stop 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f845dd9e700 1 -- 172.21.15.78:0/2573708406 shutdown_connections 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f845dd9e700 1 --2- 172.21.15.78:0/2573708406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845810ab90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f845dd9e700 1 -- 172.21.15.78:0/2573708406 >> 172.21.15.78:0/2573708406 conn(0x7f84581021b0 msgr2=0x7f84581045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f845dd9e700 1 -- 172.21.15.78:0/2573708406 shutdown_connections 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.096+0000 7f845dd9e700 1 -- 172.21.15.78:0/2573708406 wait complete. 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.097+0000 7f845dd9e700 1 Processor -- start 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.097+0000 7f845dd9e700 1 -- start start 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.097+0000 7f845dd9e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845819bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.097+0000 7f845dd9e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f845810c0d0 con 0x7f84581087a0 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.097+0000 7f84577fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845819bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f84577fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845819bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38478/0 (socket says 172.21.15.78:38478) 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f84577fe700 1 -- 172.21.15.78:0/2329760823 learned_addr learned my addr 172.21.15.78:0/2329760823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f84577fe700 1 -- 172.21.15.78:0/2329760823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845819c080 con 0x7f84581087a0 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f84577fe700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845819bb40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8448011a00 tx=0x7f8448011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f8454ff9700 1 -- 172.21.15.78:0/2329760823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84480039e0 con 0x7f84581087a0 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f8454ff9700 1 -- 172.21.15.78:0/2329760823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f844801a8e0 con 0x7f84581087a0 2024-03-16T10:57:08.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f8454ff9700 1 -- 172.21.15.78:0/2329760823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8448024760 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845819c310 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.098+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845819f000 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.099+0000 7f8454ff9700 1 -- 172.21.15.78:0/2329760823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f8448022020 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.099+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8458063900 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.100+0000 7f8454ff9700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8440038fc0 0x7f844003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.100+0000 7f8454ff9700 1 -- 172.21.15.78:0/2329760823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f844804d1d0 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.100+0000 7f8456ffd700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8440038fc0 0x7f844003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.100+0000 7f8456ffd700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8440038fc0 0x7f844003b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f844c006fd0 tx=0x7f844c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.109+0000 7f8454ff9700 1 -- 172.21.15.78:0/2329760823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8448019c70 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.328+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f845819f5b0 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.328+0000 7f8454ff9700 1 -- 172.21.15.78:0/2329760823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f844800bc40 con 0x7f84581087a0 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8440038fc0 msgr2=0x7f844003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8440038fc0 0x7f844003b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f844c006fd0 tx=0x7f844c006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 msgr2=0x7f845819bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:08.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845819bb40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8448011a00 tx=0x7f8448011ae0 comp rx=0 tx=0).stop 2024-03-16T10:57:08.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 shutdown_connections 2024-03-16T10:57:08.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8440038fc0 0x7f844003b480 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:08.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 --2- 172.21.15.78:0/2329760823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84581087a0 0x7f845819bb40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:08.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.330+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 >> 172.21.15.78:0/2329760823 conn(0x7f84581021b0 msgr2=0x7f8458102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:08.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.331+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 shutdown_connections 2024-03-16T10:57:08.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:07.331+0000 7f845dd9e700 1 -- 172.21.15.78:0/2329760823 wait complete. 2024-03-16T10:57:08.033 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:57:08.033 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-16T10:57:08.040 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:57:08.041 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-16T10:57:08.041 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-16T10:57:08.041 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-16T10:57:08.043 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph Dashboard is now available at: 2024-03-16T10:57:08.043 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:08.043 INFO:teuthology.orchestra.run.smithi078.stdout: URL: https://smithi078.front.sepia.ceph.com:8443/ 2024-03-16T10:57:08.043 INFO:teuthology.orchestra.run.smithi078.stdout: User: admin 2024-03-16T10:57:08.043 INFO:teuthology.orchestra.run.smithi078.stdout: Password: mlep10w84u 2024-03-16T10:57:08.043 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:08.043 INFO:teuthology.orchestra.run.smithi078.stdout:Saving cluster configuration to /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config directory 2024-03-16T10:57:08.044 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling autotune for osd_memory_target 2024-03-16T10:57:08.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:08 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2329760823' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-16T10:57:09.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:09 smithi078 ceph-mon[32235]: mgrmap e13: x(active, since 6s) 2024-03-16T10:57:09.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179fbf6700 1 Processor -- start 2024-03-16T10:57:09.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179fbf6700 1 -- start start 2024-03-16T10:57:09.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179fbf6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f1798107000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179fbf6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f17981075d0 con 0x7f1798106be0 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179d992700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f1798107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179d992700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f1798107000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38490/0 (socket says 172.21.15.78:38490) 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179d992700 1 -- 172.21.15.78:0/3342180663 learned_addr learned my addr 172.21.15.78:0/3342180663 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.445+0000 7f179d992700 1 -- 172.21.15.78:0/3342180663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1798107de0 con 0x7f1798106be0 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179d992700 1 --2- 172.21.15.78:0/3342180663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f1798107000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f17940098a0 tx=0x7f1794009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b00770c0b4cbeb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179c990700 1 -- 172.21.15.78:0/3342180663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1794004160 con 0x7f1798106be0 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179c990700 1 -- 172.21.15.78:0/3342180663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1794004300 con 0x7f1798106be0 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179c990700 1 -- 172.21.15.78:0/3342180663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1794011a50 con 0x7f1798106be0 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179fbf6700 1 -- 172.21.15.78:0/3342180663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 msgr2=0x7f1798107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179fbf6700 1 --2- 172.21.15.78:0/3342180663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f1798107000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f17940098a0 tx=0x7f1794009bb0 comp rx=0 tx=0).stop 2024-03-16T10:57:09.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179fbf6700 1 -- 172.21.15.78:0/3342180663 shutdown_connections 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179fbf6700 1 --2- 172.21.15.78:0/3342180663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f1798107000 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179fbf6700 1 -- 172.21.15.78:0/3342180663 >> 172.21.15.78:0/3342180663 conn(0x7f1798102200 msgr2=0x7f1798104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179fbf6700 1 -- 172.21.15.78:0/3342180663 shutdown_connections 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.446+0000 7f179fbf6700 1 -- 172.21.15.78:0/3342180663 wait complete. 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179fbf6700 1 Processor -- start 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179fbf6700 1 -- start start 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179fbf6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f179819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179fbf6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f179810b190 con 0x7f1798106be0 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179d992700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f179819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179d992700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f179819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38492/0 (socket says 172.21.15.78:38492) 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179d992700 1 -- 172.21.15.78:0/2981708518 learned_addr learned my addr 172.21.15.78:0/2981708518 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.447+0000 7f179d992700 1 -- 172.21.15.78:0/2981708518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179819c090 con 0x7f1798106be0 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.448+0000 7f179d992700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f179819bb50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1794011a20 tx=0x7f1794011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:09.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.448+0000 7f178effd700 1 -- 172.21.15.78:0/2981708518 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1794003af0 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.448+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179819c320 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.448+0000 7f178effd700 1 -- 172.21.15.78:0/2981708518 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f179401a8e0 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.448+0000 7f178effd700 1 -- 172.21.15.78:0/2981708518 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1794024860 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.448+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179819efc0 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.449+0000 7f178effd700 1 -- 172.21.15.78:0/2981708518 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f1794022020 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.449+0000 7f178effd700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1784039010 0x7f178403b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.449+0000 7f178effd700 1 -- 172.21.15.78:0/2981708518 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f179404d310 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.449+0000 7f179d191700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1784039010 0x7f178403b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.449+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1798063900 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.449+0000 7f179d191700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1784039010 0x7f178403b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1788006fd0 tx=0x7f1788006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.454+0000 7f178effd700 1 -- 172.21.15.78:0/2981708518 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1794018350 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.654+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f179802de50 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.654+0000 7f178effd700 1 -- 172.21.15.78:0/2981708518 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f1794032020 con 0x7f1798106be0 2024-03-16T10:57:09.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.656+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1784039010 msgr2=0x7f178403b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.656+0000 7f179fbf6700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1784039010 0x7f178403b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1788006fd0 tx=0x7f1788006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.656+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 msgr2=0x7f179819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.656+0000 7f179fbf6700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f179819bb50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1794011a20 tx=0x7f1794011b00 comp rx=0 tx=0).stop 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.656+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 shutdown_connections 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.656+0000 7f179fbf6700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1784039010 0x7f178403b4d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.657+0000 7f179fbf6700 1 --2- 172.21.15.78:0/2981708518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1798106be0 0x7f179819bb50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.657+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 >> 172.21.15.78:0/2981708518 conn(0x7f1798102200 msgr2=0x7f1798103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.657+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 shutdown_connections 2024-03-16T10:57:09.477 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:08.657+0000 7f179fbf6700 1 -- 172.21.15.78:0/2981708518 wait complete. 2024-03-16T10:57:11.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.877+0000 7f14227a1700 1 Processor -- start 2024-03-16T10:57:11.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.878+0000 7f14227a1700 1 -- start start 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.878+0000 7f14227a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c10a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.878+0000 7f14227a1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f141c07bdd0 con 0x7f141c1085f0 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.878+0000 7f142179f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c10a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.878+0000 7f142179f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c10a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38494/0 (socket says 172.21.15.78:38494) 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.878+0000 7f142179f700 1 -- 172.21.15.78:0/2316725776 learned_addr learned my addr 172.21.15.78:0/2316725776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.878+0000 7f142179f700 1 -- 172.21.15.78:0/2316725776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c10af20 con 0x7f141c1085f0 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f142179f700 1 --2- 172.21.15.78:0/2316725776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c10a9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f140c0098a0 tx=0x7f140c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce5fc62eaa3f9b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f1413fff700 1 -- 172.21.15.78:0/2316725776 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f140c004160 con 0x7f141c1085f0 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f1413fff700 1 -- 172.21.15.78:0/2316725776 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f140c004300 con 0x7f141c1085f0 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f1413fff700 1 -- 172.21.15.78:0/2316725776 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f140c011a50 con 0x7f141c1085f0 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f14227a1700 1 -- 172.21.15.78:0/2316725776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 msgr2=0x7f141c10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f14227a1700 1 --2- 172.21.15.78:0/2316725776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c10a9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f140c0098a0 tx=0x7f140c009bb0 comp rx=0 tx=0).stop 2024-03-16T10:57:11.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f14227a1700 1 -- 172.21.15.78:0/2316725776 shutdown_connections 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f14227a1700 1 --2- 172.21.15.78:0/2316725776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c10a9e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f14227a1700 1 -- 172.21.15.78:0/2316725776 >> 172.21.15.78:0/2316725776 conn(0x7f141c101fe0 msgr2=0x7f141c104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f14227a1700 1 -- 172.21.15.78:0/2316725776 shutdown_connections 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.879+0000 7f14227a1700 1 -- 172.21.15.78:0/2316725776 wait complete. 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f14227a1700 1 Processor -- start 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f14227a1700 1 -- start start 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f14227a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c19b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f14227a1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f141c10bed0 con 0x7f141c1085f0 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f142179f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c19b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f142179f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c19b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38510/0 (socket says 172.21.15.78:38510) 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f142179f700 1 -- 172.21.15.78:0/1700191384 learned_addr learned my addr 172.21.15.78:0/1700191384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.880+0000 7f142179f700 1 -- 172.21.15.78:0/1700191384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c19be80 con 0x7f141c1085f0 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.881+0000 7f142179f700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c19b940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f140c011a00 tx=0x7f140c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:11.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.881+0000 7f14127fc700 1 -- 172.21.15.78:0/1700191384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f140c003af0 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.881+0000 7f14127fc700 1 -- 172.21.15.78:0/1700191384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f140c01a8e0 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.881+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141c19c110 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.881+0000 7f14127fc700 1 -- 172.21.15.78:0/1700191384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f140c024840 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.881+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141c19ee00 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.882+0000 7f14127fc700 1 -- 172.21.15.78:0/1700191384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f140c022020 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.882+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141c063900 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.882+0000 7f14127fc700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1408039010 0x7f140803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.882+0000 7f14127fc700 1 -- 172.21.15.78:0/1700191384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f140c04d4b0 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.882+0000 7f1420f9e700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1408039010 0x7f140803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.882+0000 7f1420f9e700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1408039010 0x7f140803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1418006fd0 tx=0x7f1418006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:09.887+0000 7f14127fc700 1 -- 172.21.15.78:0/1700191384 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f140c019c70 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.199+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f141c19f3b0 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.201+0000 7f14127fc700 1 -- 172.21.15.78:0/1700191384 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f140c01adb0 con 0x7f141c1085f0 2024-03-16T10:57:11.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1408039010 msgr2=0x7f140803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1408039010 0x7f140803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1418006fd0 tx=0x7f1418006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 msgr2=0x7f141c19b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c19b940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f140c011a00 tx=0x7f140c011ae0 comp rx=0 tx=0).stop 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 shutdown_connections 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1408039010 0x7f140803b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 --2- 172.21.15.78:0/1700191384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f141c1085f0 0x7f141c19b940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 >> 172.21.15.78:0/1700191384 conn(0x7f141c101fe0 msgr2=0x7f141c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 shutdown_connections 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-16T10:57:10.203+0000 7f14227a1700 1 -- 172.21.15.78:0/1700191384 wait complete. 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.028 INFO:teuthology.orchestra.run.smithi078.stdout:Or, if you are only running a single cluster on this host: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: ceph telemetry on 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout:For more information see: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:11.029 INFO:teuthology.orchestra.run.smithi078.stdout:Bootstrap complete. 2024-03-16T10:57:11.075 INFO:tasks.cephadm:Fetching config... 2024-03-16T10:57:11.076 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:57:11.076 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-16T10:57:11.094 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-16T10:57:11.094 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:57:11.094 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-16T10:57:11.149 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-16T10:57:11.149 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:57:11.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-16T10:57:11.219 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-16T10:57:11.219 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:57:11.219 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-16T10:57:11.275 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-16T10:57:11.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7DORGCym24lbxU4CZMZxZiOeHQgz0UZt44eZjronrJjH3i8WiBdc5wcPPo1m3lZdoBqqy34ZNVi2wcrqubJY6P3HGewYrvtykXnlG5GdaPlvQ3bxvVUR0Bm9BgSJ4pwnMiEosb3AforEluAVn2HnGWrtzJ44GLvcC06jh+T4bgEQ9rFpYzLD7ZbJ+Sv+MN0gkR4RZBK6QTm1i97xgIeCk19qX+Nl/BjubDZti0F0eHZkfxzJHM4HORBcnbSiLa1FrKncWVHUDQhGsZEUzN766HHWJG92ad/p+NbSEp2HVa9ovSaQm+yGJvTOUIXJIGY+pmVv06TJfUCD6qyPx5X4r9DHAZkWlrCT6JzWP2OuHi+BrIHKhsSGbn2G0Tx960pIq3dDSLZkPSbcz9DpMl2coJVgITi34mcQUEpi1TzMC9X36hdBpi+WhP/wXJjPF0Tvm+bxBB3ftCqsyVPoE2Z5nU6CYGSrVHPKHBFepF3l43NPVmEtcekwmDZHXGfvNTac= ceph-a1941e5e-e383-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T10:57:11.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:11 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1700191384' entity='client.admin' 2024-03-16T10:57:11.351 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7DORGCym24lbxU4CZMZxZiOeHQgz0UZt44eZjronrJjH3i8WiBdc5wcPPo1m3lZdoBqqy34ZNVi2wcrqubJY6P3HGewYrvtykXnlG5GdaPlvQ3bxvVUR0Bm9BgSJ4pwnMiEosb3AforEluAVn2HnGWrtzJ44GLvcC06jh+T4bgEQ9rFpYzLD7ZbJ+Sv+MN0gkR4RZBK6QTm1i97xgIeCk19qX+Nl/BjubDZti0F0eHZkfxzJHM4HORBcnbSiLa1FrKncWVHUDQhGsZEUzN766HHWJG92ad/p+NbSEp2HVa9ovSaQm+yGJvTOUIXJIGY+pmVv06TJfUCD6qyPx5X4r9DHAZkWlrCT6JzWP2OuHi+BrIHKhsSGbn2G0Tx960pIq3dDSLZkPSbcz9DpMl2coJVgITi34mcQUEpi1TzMC9X36hdBpi+WhP/wXJjPF0Tvm+bxBB3ftCqsyVPoE2Z5nU6CYGSrVHPKHBFepF3l43NPVmEtcekwmDZHXGfvNTac= ceph-a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:11.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7DORGCym24lbxU4CZMZxZiOeHQgz0UZt44eZjronrJjH3i8WiBdc5wcPPo1m3lZdoBqqy34ZNVi2wcrqubJY6P3HGewYrvtykXnlG5GdaPlvQ3bxvVUR0Bm9BgSJ4pwnMiEosb3AforEluAVn2HnGWrtzJ44GLvcC06jh+T4bgEQ9rFpYzLD7ZbJ+Sv+MN0gkR4RZBK6QTm1i97xgIeCk19qX+Nl/BjubDZti0F0eHZkfxzJHM4HORBcnbSiLa1FrKncWVHUDQhGsZEUzN766HHWJG92ad/p+NbSEp2HVa9ovSaQm+yGJvTOUIXJIGY+pmVv06TJfUCD6qyPx5X4r9DHAZkWlrCT6JzWP2OuHi+BrIHKhsSGbn2G0Tx960pIq3dDSLZkPSbcz9DpMl2coJVgITi34mcQUEpi1TzMC9X36hdBpi+WhP/wXJjPF0Tvm+bxBB3ftCqsyVPoE2Z5nU6CYGSrVHPKHBFepF3l43NPVmEtcekwmDZHXGfvNTac= ceph-a1941e5e-e383-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T10:57:11.412 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7DORGCym24lbxU4CZMZxZiOeHQgz0UZt44eZjronrJjH3i8WiBdc5wcPPo1m3lZdoBqqy34ZNVi2wcrqubJY6P3HGewYrvtykXnlG5GdaPlvQ3bxvVUR0Bm9BgSJ4pwnMiEosb3AforEluAVn2HnGWrtzJ44GLvcC06jh+T4bgEQ9rFpYzLD7ZbJ+Sv+MN0gkR4RZBK6QTm1i97xgIeCk19qX+Nl/BjubDZti0F0eHZkfxzJHM4HORBcnbSiLa1FrKncWVHUDQhGsZEUzN766HHWJG92ad/p+NbSEp2HVa9ovSaQm+yGJvTOUIXJIGY+pmVv06TJfUCD6qyPx5X4r9DHAZkWlrCT6JzWP2OuHi+BrIHKhsSGbn2G0Tx960pIq3dDSLZkPSbcz9DpMl2coJVgITi34mcQUEpi1TzMC9X36hdBpi+WhP/wXJjPF0Tvm+bxBB3ftCqsyVPoE2Z5nU6CYGSrVHPKHBFepF3l43NPVmEtcekwmDZHXGfvNTac= ceph-a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:11.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7DORGCym24lbxU4CZMZxZiOeHQgz0UZt44eZjronrJjH3i8WiBdc5wcPPo1m3lZdoBqqy34ZNVi2wcrqubJY6P3HGewYrvtykXnlG5GdaPlvQ3bxvVUR0Bm9BgSJ4pwnMiEosb3AforEluAVn2HnGWrtzJ44GLvcC06jh+T4bgEQ9rFpYzLD7ZbJ+Sv+MN0gkR4RZBK6QTm1i97xgIeCk19qX+Nl/BjubDZti0F0eHZkfxzJHM4HORBcnbSiLa1FrKncWVHUDQhGsZEUzN766HHWJG92ad/p+NbSEp2HVa9ovSaQm+yGJvTOUIXJIGY+pmVv06TJfUCD6qyPx5X4r9DHAZkWlrCT6JzWP2OuHi+BrIHKhsSGbn2G0Tx960pIq3dDSLZkPSbcz9DpMl2coJVgITi34mcQUEpi1TzMC9X36hdBpi+WhP/wXJjPF0Tvm+bxBB3ftCqsyVPoE2Z5nU6CYGSrVHPKHBFepF3l43NPVmEtcekwmDZHXGfvNTac= ceph-a1941e5e-e383-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T10:57:11.472 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7DORGCym24lbxU4CZMZxZiOeHQgz0UZt44eZjronrJjH3i8WiBdc5wcPPo1m3lZdoBqqy34ZNVi2wcrqubJY6P3HGewYrvtykXnlG5GdaPlvQ3bxvVUR0Bm9BgSJ4pwnMiEosb3AforEluAVn2HnGWrtzJ44GLvcC06jh+T4bgEQ9rFpYzLD7ZbJ+Sv+MN0gkR4RZBK6QTm1i97xgIeCk19qX+Nl/BjubDZti0F0eHZkfxzJHM4HORBcnbSiLa1FrKncWVHUDQhGsZEUzN766HHWJG92ad/p+NbSEp2HVa9ovSaQm+yGJvTOUIXJIGY+pmVv06TJfUCD6qyPx5X4r9DHAZkWlrCT6JzWP2OuHi+BrIHKhsSGbn2G0Tx960pIq3dDSLZkPSbcz9DpMl2coJVgITi34mcQUEpi1TzMC9X36hdBpi+WhP/wXJjPF0Tvm+bxBB3ftCqsyVPoE2Z5nU6CYGSrVHPKHBFepF3l43NPVmEtcekwmDZHXGfvNTac= ceph-a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:11.487 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-16T10:57:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.021+0000 7f207090f700 1 -- 172.21.15.78:0/32888440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 msgr2=0x7f206c102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.021+0000 7f207090f700 1 --2- 172.21.15.78:0/32888440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c102640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f205c009910 tx=0x7f205c009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.022+0000 7f207090f700 1 -- 172.21.15.78:0/32888440 shutdown_connections 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.022+0000 7f207090f700 1 --2- 172.21.15.78:0/32888440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c102640 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.022+0000 7f207090f700 1 -- 172.21.15.78:0/32888440 >> 172.21.15.78:0/32888440 conn(0x7f206c0fb930 msgr2=0x7f206c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.022+0000 7f207090f700 1 -- 172.21.15.78:0/32888440 shutdown_connections 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.022+0000 7f207090f700 1 -- 172.21.15.78:0/32888440 wait complete. 2024-03-16T10:57:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.022+0000 7f207090f700 1 Processor -- start 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.022+0000 7f207090f700 1 -- start start 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f207090f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f207090f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f206c109d90 con 0x7f206c102220 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f206a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f206a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38520/0 (socket says 172.21.15.78:38520) 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f206a59c700 1 -- 172.21.15.78:0/3702728909 learned_addr learned my addr 172.21.15.78:0/3702728909 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f206a59c700 1 -- 172.21.15.78:0/3702728909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c1999d0 con 0x7f206c102220 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f206a59c700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c199490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f205c005800 tx=0x7f205c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f20637fe700 1 -- 172.21.15.78:0/3702728909 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f205c027070 con 0x7f206c102220 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f20637fe700 1 -- 172.21.15.78:0/3702728909 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f205c00be10 con 0x7f206c102220 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c199cc0 con 0x7f206c102220 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f20637fe700 1 -- 172.21.15.78:0/3702728909 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f205c00f460 con 0x7f206c102220 2024-03-16T10:57:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.023+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c19c950 con 0x7f206c102220 2024-03-16T10:57:14.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.024+0000 7f20637fe700 1 -- 172.21.15.78:0/3702728909 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f205c02a070 con 0x7f206c102220 2024-03-16T10:57:14.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.024+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f206c10a510 con 0x7f206c102220 2024-03-16T10:57:14.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.024+0000 7f20637fe700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2058039020 0x7f205803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.024+0000 7f20637fe700 1 -- 172.21.15.78:0/3702728909 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f205c01e030 con 0x7f206c102220 2024-03-16T10:57:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.024+0000 7f2063fff700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2058039020 0x7f205803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.025+0000 7f2063fff700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2058039020 0x7f205803b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2054006fd0 tx=0x7f2054006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:14.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.029+0000 7f20637fe700 1 -- 172.21.15.78:0/3702728909 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f205c013070 con 0x7f206c102220 2024-03-16T10:57:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.223+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f206c02dcf0 con 0x7f206c102220 2024-03-16T10:57:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.225+0000 7f20637fe700 1 -- 172.21.15.78:0/3702728909 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f205c02a360 con 0x7f206c102220 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2058039020 msgr2=0x7f205803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2058039020 0x7f205803b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2054006fd0 tx=0x7f2054006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 msgr2=0x7f206c199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c199490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f205c005800 tx=0x7f205c009f60 comp rx=0 tx=0).stop 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 shutdown_connections 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2058039020 0x7f205803b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 --2- 172.21.15.78:0/3702728909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f206c102220 0x7f206c199490 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 >> 172.21.15.78:0/3702728909 conn(0x7f206c0fb930 msgr2=0x7f206c0fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 shutdown_connections 2024-03-16T10:57:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:14.228+0000 7f207090f700 1 -- 172.21.15.78:0/3702728909 wait complete. 2024-03-16T10:57:15.002 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-16T10:57:15.003 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-16T10:57:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:15 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3702728909' entity='client.admin' 2024-03-16T10:57:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:57:16.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:16.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:16.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:57:16.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:16.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:16.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.296+0000 7f69b83af700 1 -- 172.21.15.78:0/2482291068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 msgr2=0x7f69b0102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.296+0000 7f69b83af700 1 --2- 172.21.15.78:0/2482291068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0102620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009910 tx=0x7f69ac009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 -- 172.21.15.78:0/2482291068 shutdown_connections 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 --2- 172.21.15.78:0/2482291068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0102620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 -- 172.21.15.78:0/2482291068 >> 172.21.15.78:0/2482291068 conn(0x7f69b00fb960 msgr2=0x7f69b00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 -- 172.21.15.78:0/2482291068 shutdown_connections 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 -- 172.21.15.78:0/2482291068 wait complete. 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 Processor -- start 2024-03-16T10:57:17.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 -- start start 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0100820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b83af700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f69b0106b60 con 0x7f69b0102200 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b614b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0100820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b614b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0100820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42318/0 (socket says 172.21.15.78:42318) 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.297+0000 7f69b614b700 1 -- 172.21.15.78:0/3568988786 learned_addr learned my addr 172.21.15.78:0/3568988786 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.298+0000 7f69b614b700 1 -- 172.21.15.78:0/3568988786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b019a0a0 con 0x7f69b0102200 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.298+0000 7f69b614b700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0100820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69ac005950 tx=0x7f69ac004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.298+0000 7f69a77fe700 1 -- 172.21.15.78:0/3568988786 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69ac026070 con 0x7f69b0102200 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.298+0000 7f69a77fe700 1 -- 172.21.15.78:0/3568988786 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69ac00bc00 con 0x7f69b0102200 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.298+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b019a270 con 0x7f69b0102200 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.298+0000 7f69a77fe700 1 -- 172.21.15.78:0/3568988786 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69ac0039a0 con 0x7f69b0102200 2024-03-16T10:57:17.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.298+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b019a610 con 0x7f69b0102200 2024-03-16T10:57:17.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.299+0000 7f69a77fe700 1 -- 172.21.15.78:0/3568988786 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f69ac029070 con 0x7f69b0102200 2024-03-16T10:57:17.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.299+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b01072e0 con 0x7f69b0102200 2024-03-16T10:57:17.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.299+0000 7f69a77fe700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f699c039020 0x7f699c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:17.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.299+0000 7f69a77fe700 1 -- 172.21.15.78:0/3568988786 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f69ac01e070 con 0x7f69b0102200 2024-03-16T10:57:17.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.299+0000 7f69b594a700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f699c039020 0x7f699c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:17.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.300+0000 7f69b594a700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f699c039020 0x7f699c03b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f69a0006fd0 tx=0x7f69a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:17.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.304+0000 7f69a77fe700 1 -- 172.21.15.78:0/3568988786 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69ac013070 con 0x7f69b0102200 2024-03-16T10:57:17.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.503+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f69b002dd20 con 0x7f699c039020 2024-03-16T10:57:17.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.506+0000 7f69a77fe700 1 -- 172.21.15.78:0/3568988786 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f69b002dd20 con 0x7f699c039020 2024-03-16T10:57:17.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.508+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f699c039020 msgr2=0x7f699c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f699c039020 0x7f699c03b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f69a0006fd0 tx=0x7f69a0006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 msgr2=0x7f69b0100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0100820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69ac005950 tx=0x7f69ac004fa0 comp rx=0 tx=0).stop 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 shutdown_connections 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f699c039020 0x7f699c03b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 --2- 172.21.15.78:0/3568988786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b0102200 0x7f69b0100820 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 >> 172.21.15.78:0/3568988786 conn(0x7f69b00fb960 msgr2=0x7f69b00fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 shutdown_connections 2024-03-16T10:57:17.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:17.509+0000 7f69b83af700 1 -- 172.21.15.78:0/3568988786 wait complete. 2024-03-16T10:57:18.273 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi098 2024-03-16T10:57:18.274 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:57:18.274 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.conf 2024-03-16T10:57:18.292 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:57:18.292 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:18.348 INFO:tasks.cephadm:Adding host smithi098 to orchestrator... 2024-03-16T10:57:18.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch host add smithi098 2024-03-16T10:57:18.588 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:57:18.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: Updating smithi078:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:18.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:19 smithi078 ceph-mon[32235]: Updating smithi078:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.090+0000 7f45ea4e5700 1 -- 172.21.15.78:0/796060487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 msgr2=0x7f45e41011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.090+0000 7f45ea4e5700 1 --2- 172.21.15.78:0/796060487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e41011f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009910 tx=0x7f45d4009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.090+0000 7f45ea4e5700 1 -- 172.21.15.78:0/796060487 shutdown_connections 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.090+0000 7f45ea4e5700 1 --2- 172.21.15.78:0/796060487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e41011f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.090+0000 7f45ea4e5700 1 -- 172.21.15.78:0/796060487 >> 172.21.15.78:0/796060487 conn(0x7f45e40fc6c0 msgr2=0x7f45e40feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.090+0000 7f45ea4e5700 1 -- 172.21.15.78:0/796060487 shutdown_connections 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.090+0000 7f45ea4e5700 1 -- 172.21.15.78:0/796060487 wait complete. 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.091+0000 7f45ea4e5700 1 Processor -- start 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.091+0000 7f45ea4e5700 1 -- start start 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.091+0000 7f45ea4e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e4198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.091+0000 7f45ea4e5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f45e410b090 con 0x7f45e4100e10 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.091+0000 7f45e3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e4198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.091+0000 7f45e3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e4198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42332/0 (socket says 172.21.15.78:42332) 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.091+0000 7f45e3fff700 1 -- 172.21.15.78:0/3123969206 learned_addr learned my addr 172.21.15.78:0/3123969206 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.092+0000 7f45e3fff700 1 -- 172.21.15.78:0/3123969206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e4198ab0 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.092+0000 7f45e3fff700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e4198570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45d40056d0 tx=0x7f45d40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.092+0000 7f45e17fa700 1 -- 172.21.15.78:0/3123969206 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45d4026070 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.092+0000 7f45e17fa700 1 -- 172.21.15.78:0/3123969206 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45d400bba0 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.092+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e4198da0 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.092+0000 7f45e17fa700 1 -- 172.21.15.78:0/3123969206 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45d40037f0 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.092+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e4199260 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.093+0000 7f45e17fa700 1 -- 172.21.15.78:0/3123969206 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f45d4029070 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.093+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e41022f0 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.093+0000 7f45e17fa700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45cc038fc0 0x7f45cc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.093+0000 7f45e17fa700 1 -- 172.21.15.78:0/3123969206 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f45d401e070 con 0x7f45e4100e10 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.093+0000 7f45e37fe700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45cc038fc0 0x7f45cc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.093+0000 7f45e37fe700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45cc038fc0 0x7f45cc03b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f45d8006fd0 tx=0x7f45d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.098+0000 7f45e17fa700 1 -- 172.21.15.78:0/3123969206 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f45d4013070 con 0x7f45e4100e10 2024-03-16T10:57:20.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:20.298+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}) -- 0x7f45e406a720 con 0x7f45cc038fc0 2024-03-16T10:57:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:21 smithi078 ceph-mon[32235]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:21 smithi078 ceph-mon[32235]: Deploying cephadm binary to smithi098 2024-03-16T10:57:22.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:22 smithi078 ceph-mon[32235]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:24.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.650+0000 7f45e17fa700 1 -- 172.21.15.78:0/3123969206 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f45e406a720 con 0x7f45cc038fc0 2024-03-16T10:57:24.651 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi098' with addr '172.21.15.98' 2024-03-16T10:57:24.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45cc038fc0 msgr2=0x7f45cc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45cc038fc0 0x7f45cc03b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f45d8006fd0 tx=0x7f45d8006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 msgr2=0x7f45e4198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e4198570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45d40056d0 tx=0x7f45d40057b0 comp rx=0 tx=0).stop 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 shutdown_connections 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45cc038fc0 0x7f45cc03b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 --2- 172.21.15.78:0/3123969206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45e4100e10 0x7f45e4198570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 >> 172.21.15.78:0/3123969206 conn(0x7f45e40fc6c0 msgr2=0x7f45e41027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 shutdown_connections 2024-03-16T10:57:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:24.653+0000 7f45ea4e5700 1 -- 172.21.15.78:0/3123969206 wait complete. 2024-03-16T10:57:25.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:24 smithi078 ceph-mon[32235]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:25.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:25.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:25.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T10:57:25.488 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:57:26.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:25 smithi078 ceph-mon[32235]: Added host smithi098 2024-03-16T10:57:26.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:26.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:27.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.034+0000 7f59b85f8700 1 -- 172.21.15.78:0/725424185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 msgr2=0x7f59b01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:27.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.034+0000 7f59b85f8700 1 --2- 172.21.15.78:0/725424185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b01011f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59a0009910 tx=0x7f59a0009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:27.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.034+0000 7f59b85f8700 1 -- 172.21.15.78:0/725424185 shutdown_connections 2024-03-16T10:57:27.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.034+0000 7f59b85f8700 1 --2- 172.21.15.78:0/725424185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b01011f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:27.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.034+0000 7f59b85f8700 1 -- 172.21.15.78:0/725424185 >> 172.21.15.78:0/725424185 conn(0x7f59b00fc6c0 msgr2=0x7f59b00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:27.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.034+0000 7f59b85f8700 1 -- 172.21.15.78:0/725424185 shutdown_connections 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b85f8700 1 -- 172.21.15.78:0/725424185 wait complete. 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b85f8700 1 Processor -- start 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b85f8700 1 -- start start 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b85f8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b0196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b85f8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f59b010b090 con 0x7f59b0100e10 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b6394700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b0196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b6394700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b0196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37494/0 (socket says 172.21.15.78:37494) 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.035+0000 7f59b6394700 1 -- 172.21.15.78:0/2747467112 learned_addr learned my addr 172.21.15.78:0/2747467112 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.036+0000 7f59b6394700 1 -- 172.21.15.78:0/2747467112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b006af50 con 0x7f59b0100e10 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.036+0000 7f59b6394700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b0196350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f59a00056d0 tx=0x7f59a00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.036+0000 7f59a77fe700 1 -- 172.21.15.78:0/2747467112 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59a0026070 con 0x7f59b0100e10 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.036+0000 7f59a77fe700 1 -- 172.21.15.78:0/2747467112 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59a000bba0 con 0x7f59b0100e10 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.036+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b0196890 con 0x7f59b0100e10 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.036+0000 7f59a77fe700 1 -- 172.21.15.78:0/2747467112 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59a00037f0 con 0x7f59b0100e10 2024-03-16T10:57:27.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.036+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b0196d50 con 0x7f59b0100e10 2024-03-16T10:57:27.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.037+0000 7f59a77fe700 1 -- 172.21.15.78:0/2747467112 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f59a0029070 con 0x7f59b0100e10 2024-03-16T10:57:27.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.037+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b01022f0 con 0x7f59b0100e10 2024-03-16T10:57:27.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.037+0000 7f59a77fe700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f599c039020 0x7f599c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:27.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.037+0000 7f59a77fe700 1 -- 172.21.15.78:0/2747467112 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f59a001e070 con 0x7f59b0100e10 2024-03-16T10:57:27.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.037+0000 7f59b5b93700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f599c039020 0x7f599c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:27.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.037+0000 7f59b5b93700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f599c039020 0x7f599c03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006fd0 tx=0x7f59ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:27.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.042+0000 7f59a77fe700 1 -- 172.21.15.78:0/2747467112 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59a0013070 con 0x7f59b0100e10 2024-03-16T10:57:27.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:26 smithi078 ceph-mon[32235]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:27.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:27.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.244+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f59b006a7b0 con 0x7f599c039020 2024-03-16T10:57:27.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.245+0000 7f59a77fe700 1 -- 172.21.15.78:0/2747467112 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f59b006a7b0 con 0x7f599c039020 2024-03-16T10:57:27.246 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:27.246 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}] 2024-03-16T10:57:27.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f599c039020 msgr2=0x7f599c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:27.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f599c039020 0x7f599c03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006fd0 tx=0x7f59ac006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:27.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 msgr2=0x7f59b0196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:27.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b0196350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f59a00056d0 tx=0x7f59a00057b0 comp rx=0 tx=0).stop 2024-03-16T10:57:27.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 shutdown_connections 2024-03-16T10:57:27.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f599c039020 0x7f599c03b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:27.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 --2- 172.21.15.78:0/2747467112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59b0100e10 0x7f59b0196350 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:27.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 >> 172.21.15.78:0/2747467112 conn(0x7f59b00fc6c0 msgr2=0x7f59b01027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:27.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 shutdown_connections 2024-03-16T10:57:27.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:27.247+0000 7f59b85f8700 1 -- 172.21.15.78:0/2747467112 wait complete. 2024-03-16T10:57:27.831 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-03-16T10:57:27.831 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:57:27.831 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-03-16T10:57:27.849 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:57:27.849 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:27.907 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-03-16T10:57:27.907 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch host add smithi153 2024-03-16T10:57:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:57:29.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:28 smithi078 ceph-mon[32235]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:57:29.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:28 smithi078 ceph-mon[32235]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.565+0000 7f97f158a700 1 -- 172.21.15.78:0/2452261056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 msgr2=0x7f97ec102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.565+0000 7f97f158a700 1 --2- 172.21.15.78:0/2452261056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec102320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009910 tx=0x7f97dc009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.565+0000 7f97f158a700 1 -- 172.21.15.78:0/2452261056 shutdown_connections 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.565+0000 7f97f158a700 1 --2- 172.21.15.78:0/2452261056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec102320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.565+0000 7f97f158a700 1 -- 172.21.15.78:0/2452261056 >> 172.21.15.78:0/2452261056 conn(0x7f97ec0fd7f0 msgr2=0x7f97ec0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.565+0000 7f97f158a700 1 -- 172.21.15.78:0/2452261056 shutdown_connections 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.565+0000 7f97f158a700 1 -- 172.21.15.78:0/2452261056 wait complete. 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.566+0000 7f97f158a700 1 Processor -- start 2024-03-16T10:57:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.566+0000 7f97f158a700 1 -- start start 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.566+0000 7f97f158a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.566+0000 7f97f158a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec10c1c0 con 0x7f97ec101f40 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.566+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.566+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37516/0 (socket says 172.21.15.78:37516) 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.566+0000 7f97eaffd700 1 -- 172.21.15.78:0/4275924677 learned_addr learned my addr 172.21.15.78:0/4275924677 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.567+0000 7f97eaffd700 1 -- 172.21.15.78:0/4275924677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec197c70 con 0x7f97ec101f40 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.567+0000 7f97eaffd700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec197730 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0056d0 tx=0x7f97dc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:29.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.567+0000 7f97d3fff700 1 -- 172.21.15.78:0/4275924677 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97dc026070 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.567+0000 7f97d3fff700 1 -- 172.21.15.78:0/4275924677 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97dc00bba0 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.567+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ec197f60 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.567+0000 7f97d3fff700 1 -- 172.21.15.78:0/4275924677 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97dc0037f0 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.567+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ec19c000 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.568+0000 7f97d3fff700 1 -- 172.21.15.78:0/4275924677 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f97dc029070 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.568+0000 7f97d3fff700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97d4038fc0 0x7f97d403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.568+0000 7f97d3fff700 1 -- 172.21.15.78:0/4275924677 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f97dc01e070 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.568+0000 7f97ea7fc700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97d4038fc0 0x7f97d403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.568+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ec103420 con 0x7f97ec101f40 2024-03-16T10:57:29.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.568+0000 7f97ea7fc700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97d4038fc0 0x7f97d403b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006fd0 tx=0x7f97e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:29.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.573+0000 7f97d3fff700 1 -- 172.21.15.78:0/4275924677 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97dc030070 con 0x7f97ec101f40 2024-03-16T10:57:29.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:29.770+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) -- 0x7f97ec063960 con 0x7f97d4038fc0 2024-03-16T10:57:31.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:31.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:31.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:31.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:31.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:31 smithi078 ceph-mon[32235]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:31 smithi078 ceph-mon[32235]: Deploying cephadm binary to smithi153 2024-03-16T10:57:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:31 smithi078 ceph-mon[32235]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:57:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:32 smithi078 ceph-mon[32235]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:34.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.075+0000 7f97d3fff700 1 -- 172.21.15.78:0/4275924677 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f97ec063960 con 0x7f97d4038fc0 2024-03-16T10:57:34.076 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97d4038fc0 msgr2=0x7f97d403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97d4038fc0 0x7f97d403b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006fd0 tx=0x7f97e0006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 msgr2=0x7f97ec197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec197730 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0056d0 tx=0x7f97dc0057b0 comp rx=0 tx=0).stop 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 shutdown_connections 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97d4038fc0 0x7f97d403b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 --2- 172.21.15.78:0/4275924677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97ec101f40 0x7f97ec197730 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 >> 172.21.15.78:0/4275924677 conn(0x7f97ec0fd7f0 msgr2=0x7f97ec103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 shutdown_connections 2024-03-16T10:57:34.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:34.077+0000 7f97f158a700 1 -- 172.21.15.78:0/4275924677 wait complete. 2024-03-16T10:57:34.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T10:57:34.928 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:57:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:35 smithi078 ceph-mon[32235]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:35 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:35 smithi078 ceph-mon[32235]: Added host smithi153 2024-03-16T10:57:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:35 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:35 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:36.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.472+0000 7f4626395700 1 -- 172.21.15.78:0/4063414879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 msgr2=0x7f46201070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.472+0000 7f4626395700 1 --2- 172.21.15.78:0/4063414879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201070c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4608009910 tx=0x7f4608009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.472+0000 7f4626395700 1 -- 172.21.15.78:0/4063414879 shutdown_connections 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.472+0000 7f4626395700 1 --2- 172.21.15.78:0/4063414879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201070c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.472+0000 7f4626395700 1 -- 172.21.15.78:0/4063414879 >> 172.21.15.78:0/4063414879 conn(0x7f46200fbe20 msgr2=0x7f46200fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.472+0000 7f4626395700 1 -- 172.21.15.78:0/4063414879 shutdown_connections 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.472+0000 7f4626395700 1 -- 172.21.15.78:0/4063414879 wait complete. 2024-03-16T10:57:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f4626395700 1 Processor -- start 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f4626395700 1 -- start start 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f4626395700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f4626395700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f462010c2f0 con 0x7f4620106ce0 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f461ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f461ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38000/0 (socket says 172.21.15.78:38000) 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f461ffff700 1 -- 172.21.15.78:0/1175164070 learned_addr learned my addr 172.21.15.78:0/1175164070 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.473+0000 7f461ffff700 1 -- 172.21.15.78:0/1175164070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4620102f20 con 0x7f4620106ce0 2024-03-16T10:57:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.474+0000 7f461ffff700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201012e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46080056d0 tx=0x7f46080057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.474+0000 7f461d7fa700 1 -- 172.21.15.78:0/1175164070 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4608026070 con 0x7f4620106ce0 2024-03-16T10:57:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.474+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46201031b0 con 0x7f4620106ce0 2024-03-16T10:57:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.474+0000 7f461d7fa700 1 -- 172.21.15.78:0/1175164070 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f460800bc00 con 0x7f4620106ce0 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.474+0000 7f461d7fa700 1 -- 172.21.15.78:0/1175164070 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46080039b0 con 0x7f4620106ce0 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.474+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46201019f0 con 0x7f4620106ce0 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.475+0000 7f461d7fa700 1 -- 172.21.15.78:0/1175164070 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4608029070 con 0x7f4620106ce0 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.475+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4620063900 con 0x7f4620106ce0 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.475+0000 7f461d7fa700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f460c038fc0 0x7f460c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.475+0000 7f461d7fa700 1 -- 172.21.15.78:0/1175164070 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f460801e070 con 0x7f4620106ce0 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.475+0000 7f461f7fe700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f460c038fc0 0x7f460c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.475+0000 7f461f7fe700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f460c038fc0 0x7f460c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4610006fd0 tx=0x7f4610006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.480+0000 7f461d7fa700 1 -- 172.21.15.78:0/1175164070 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4608013070 con 0x7f4620106ce0 2024-03-16T10:57:36.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.688+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4620103380 con 0x7f460c038fc0 2024-03-16T10:57:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.690+0000 7f461d7fa700 1 -- 172.21.15.78:0/1175164070 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f4620103380 con 0x7f460c038fc0 2024-03-16T10:57:36.690 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T10:57:36.690 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f460c038fc0 msgr2=0x7f460c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f460c038fc0 0x7f460c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4610006fd0 tx=0x7f4610006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 msgr2=0x7f46201012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201012e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46080056d0 tx=0x7f46080057b0 comp rx=0 tx=0).stop 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 shutdown_connections 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f460c038fc0 0x7f460c03b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 --2- 172.21.15.78:0/1175164070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4620106ce0 0x7f46201012e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 >> 172.21.15.78:0/1175164070 conn(0x7f46200fbe20 msgr2=0x7f46200fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 shutdown_connections 2024-03-16T10:57:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:36.692+0000 7f4626395700 1 -- 172.21.15.78:0/1175164070 wait complete. 2024-03-16T10:57:37.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:37 smithi078 ceph-mon[32235]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:37.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:37 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:37.327 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-16T10:57:37.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-16T10:57:37.563 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:57:38.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:38 smithi078 ceph-mon[32235]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:57:39.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.085+0000 7f881ccbf700 1 -- 172.21.15.78:0/1980652623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 msgr2=0x7f88181011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:39.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.085+0000 7f881ccbf700 1 --2- 172.21.15.78:0/1980652623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f88181011f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f880c009910 tx=0x7f880c009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:39.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.085+0000 7f881ccbf700 1 -- 172.21.15.78:0/1980652623 shutdown_connections 2024-03-16T10:57:39.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.085+0000 7f881ccbf700 1 --2- 172.21.15.78:0/1980652623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f88181011f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:39.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.085+0000 7f881ccbf700 1 -- 172.21.15.78:0/1980652623 >> 172.21.15.78:0/1980652623 conn(0x7f88180fc6c0 msgr2=0x7f88180feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:39.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.085+0000 7f881ccbf700 1 -- 172.21.15.78:0/1980652623 shutdown_connections 2024-03-16T10:57:39.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.085+0000 7f881ccbf700 1 -- 172.21.15.78:0/1980652623 wait complete. 2024-03-16T10:57:39.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.086+0000 7f881ccbf700 1 Processor -- start 2024-03-16T10:57:39.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.086+0000 7f881ccbf700 1 -- start start 2024-03-16T10:57:39.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.086+0000 7f881ccbf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f8818196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:39.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.086+0000 7f881ccbf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f881810b090 con 0x7f8818100e10 2024-03-16T10:57:39.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.086+0000 7f881659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f8818196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:39.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.086+0000 7f881659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f8818196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38008/0 (socket says 172.21.15.78:38008) 2024-03-16T10:57:39.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.086+0000 7f881659c700 1 -- 172.21.15.78:0/3870408618 learned_addr learned my addr 172.21.15.78:0/3870408618 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:57:39.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.087+0000 7f881659c700 1 -- 172.21.15.78:0/3870408618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8818196b60 con 0x7f8818100e10 2024-03-16T10:57:39.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.087+0000 7f881659c700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f8818196620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f880c0056d0 tx=0x7f880c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.087+0000 7f88077fe700 1 -- 172.21.15.78:0/3870408618 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f880c026070 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.087+0000 7f88077fe700 1 -- 172.21.15.78:0/3870408618 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f880c00bba0 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.087+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8818196e50 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.087+0000 7f88077fe700 1 -- 172.21.15.78:0/3870408618 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f880c0037f0 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.087+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881819aef0 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.088+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88181022f0 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.088+0000 7f88077fe700 1 -- 172.21.15.78:0/3870408618 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f880c029070 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.088+0000 7f88077fe700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8800039020 0x7f880003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.088+0000 7f88077fe700 1 -- 172.21.15.78:0/3870408618 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f880c01e070 con 0x7f8818100e10 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.088+0000 7f8815d9b700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8800039020 0x7f880003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:39.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.089+0000 7f8815d9b700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8800039020 0x7f880003b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8808006fd0 tx=0x7f8808006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:39.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.093+0000 7f88077fe700 1 -- 172.21.15.78:0/3870408618 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f880c013070 con 0x7f8818100e10 2024-03-16T10:57:39.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:39.292+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f8818197980 con 0x7f8818100e10 2024-03-16T10:57:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:39 smithi078 ceph-mon[32235]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.110+0000 7f88077fe700 1 -- 172.21.15.78:0/3870408618 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f880c030020 con 0x7f8818100e10 2024-03-16T10:57:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8800039020 msgr2=0x7f880003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8800039020 0x7f880003b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8808006fd0 tx=0x7f8808006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 msgr2=0x7f8818196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f8818196620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f880c0056d0 tx=0x7f880c0057b0 comp rx=0 tx=0).stop 2024-03-16T10:57:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 shutdown_connections 2024-03-16T10:57:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8800039020 0x7f880003b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 --2- 172.21.15.78:0/3870408618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8818100e10 0x7f8818196620 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 >> 172.21.15.78:0/3870408618 conn(0x7f88180fc6c0 msgr2=0x7f88181027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 shutdown_connections 2024-03-16T10:57:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:57:40.112+0000 7f881ccbf700 1 -- 172.21.15.78:0/3870408618 wait complete. 2024-03-16T10:57:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-03-16T10:57:40.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3870408618' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T10:57:40.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:40.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:40.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:40.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:40.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:57:40.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:40.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:40.811 INFO:tasks.cephadm:Adding mon.a on smithi078 2024-03-16T10:57:40.811 INFO:tasks.cephadm:Adding mon.b on smithi098 2024-03-16T10:57:40.811 INFO:tasks.cephadm:Adding mon.c on smithi153 2024-03-16T10:57:40.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c' 2024-03-16T10:57:41.036 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T10:57:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:41.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3870408618' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T10:57:41.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:57:41.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:41.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:41.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:42.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:42 smithi078 ceph-mon[32235]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:42.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:42 smithi078 ceph-mon[32235]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:57:42.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.586+0000 7fea4e5b4700 1 -- 172.21.15.153:0/1784632846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 msgr2=0x7fea48103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:42.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.586+0000 7fea4e5b4700 1 --2- 172.21.15.153:0/1784632846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48103240 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea38009910 tx=0x7fea38009c20 comp rx=0 tx=0).stop 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.587+0000 7fea4e5b4700 1 -- 172.21.15.153:0/1784632846 shutdown_connections 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.587+0000 7fea4e5b4700 1 --2- 172.21.15.153:0/1784632846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48103240 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.587+0000 7fea4e5b4700 1 -- 172.21.15.153:0/1784632846 >> 172.21.15.153:0/1784632846 conn(0x7fea48077810 msgr2=0x7fea480feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.587+0000 7fea4e5b4700 1 -- 172.21.15.153:0/1784632846 shutdown_connections 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.587+0000 7fea4e5b4700 1 -- 172.21.15.153:0/1784632846 wait complete. 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.587+0000 7fea4e5b4700 1 Processor -- start 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.587+0000 7fea4e5b4700 1 -- start start 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea4e5b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:42.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea4e5b4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea4810c200 con 0x7fea48102e60 2024-03-16T10:57:42.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:42.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.153:58910/0 (socket says 172.21.15.153:58910) 2024-03-16T10:57:42.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea47fff700 1 -- 172.21.15.153:0/2245443038 learned_addr learned my addr 172.21.15.153:0/2245443038 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea47fff700 1 -- 172.21.15.153:0/2245443038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea481029b0 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea47fff700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48100ac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea380056d0 tx=0x7fea380057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.588+0000 7fea457fa700 1 -- 172.21.15.153:0/2245443038 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea38026070 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.589+0000 7fea457fa700 1 -- 172.21.15.153:0/2245443038 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea38004370 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.589+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea48101000 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.589+0000 7fea457fa700 1 -- 172.21.15.153:0/2245443038 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea380039b0 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.589+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea481014c0 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.589+0000 7fea457fa700 1 -- 172.21.15.153:0/2245443038 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fea38029070 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.589+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea48069fa0 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.590+0000 7fea457fa700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fea30039020 0x7fea3003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.590+0000 7fea457fa700 1 -- 172.21.15.153:0/2245443038 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fea3801e070 con 0x7fea48102e60 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.590+0000 7fea477fe700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fea30039020 0x7fea3003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.590+0000 7fea477fe700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fea30039020 0x7fea3003b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea3c006fd0 tx=0x7fea3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:42.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.595+0000 7fea457fa700 1 -- 172.21.15.153:0/2245443038 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea38013070 con 0x7fea48102e60 2024-03-16T10:57:42.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.800+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}) -- 0x7fea481016f0 con 0x7fea30039020 2024-03-16T10:57:42.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.807+0000 7fea457fa700 1 -- 172.21.15.153:0/2245443038 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fea481016f0 con 0x7fea30039020 2024-03-16T10:57:42.807 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-03-16T10:57:42.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fea30039020 msgr2=0x7fea3003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fea30039020 0x7fea3003b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea3c006fd0 tx=0x7fea3c006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 msgr2=0x7fea48100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48100ac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea380056d0 tx=0x7fea380057b0 comp rx=0 tx=0).stop 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 shutdown_connections 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fea30039020 0x7fea3003b4e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 --2- 172.21.15.153:0/2245443038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea48102e60 0x7fea48100ac0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 >> 172.21.15.153:0/2245443038 conn(0x7fea48077810 msgr2=0x7fea48105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 shutdown_connections 2024-03-16T10:57:42.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:42.809+0000 7fea4e5b4700 1 -- 172.21.15.153:0/2245443038 wait complete. 2024-03-16T10:57:43.364 DEBUG:teuthology.orchestra.run.smithi098:mon.b> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.b.service 2024-03-16T10:57:43.367 DEBUG:teuthology.orchestra.run.smithi153:mon.c> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.c.service 2024-03-16T10:57:43.370 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T10:57:43.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T10:57:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c;count:3 2024-03-16T10:57:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:43.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:43.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:43.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:43.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:43.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:57:43.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:43.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:43 smithi078 ceph-mon[32235]: Deploying daemon mon.c on smithi153 2024-03-16T10:57:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:44 smithi078 ceph-mon[32235]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:44.974 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:46.962 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:46 smithi153 podman[35461]: 2024-03-16 10:57:46.855034614 +0000 UTC m=+0.218256505 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.061+0000 7f5f8e090700 1 -- 172.21.15.153:0/2874866902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 msgr2=0x7f5f88100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.061+0000 7f5f8e090700 1 --2- 172.21.15.153:0/2874866902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88100860 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000b0a0 tx=0x7f5f8000b3b0 comp rx=0 tx=0).stop 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.061+0000 7f5f8e090700 1 -- 172.21.15.153:0/2874866902 shutdown_connections 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.061+0000 7f5f8e090700 1 --2- 172.21.15.153:0/2874866902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88100860 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.061+0000 7f5f8e090700 1 -- 172.21.15.153:0/2874866902 >> 172.21.15.153:0/2874866902 conn(0x7f5f880fbe20 msgr2=0x7f5f880fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.061+0000 7f5f8e090700 1 -- 172.21.15.153:0/2874866902 shutdown_connections 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.062+0000 7f5f8e090700 1 -- 172.21.15.153:0/2874866902 wait complete. 2024-03-16T10:57:47.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.062+0000 7f5f8e090700 1 Processor -- start 2024-03-16T10:57:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.062+0000 7f5f8e090700 1 -- start start 2024-03-16T10:57:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.062+0000 7f5f8e090700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.062+0000 7f5f8e090700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8810c190 con 0x7f5f88100480 2024-03-16T10:57:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88101400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.153:48278/0 (socket says 172.21.15.153:48278) 2024-03-16T10:57:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f877fe700 1 -- 172.21.15.153:0/1521931189 learned_addr learned my addr 172.21.15.153:0/1521931189 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T10:57:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f877fe700 1 -- 172.21.15.153:0/1521931189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f881032f0 con 0x7f5f88100480 2024-03-16T10:57:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f877fe700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88101400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f800076a0 tx=0x7f5f80003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f84ff9700 1 -- 172.21.15.153:0/1521931189 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f80017040 con 0x7f5f88100480 2024-03-16T10:57:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f84ff9700 1 -- 172.21.15.153:0/1521931189 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f80022070 con 0x7f5f88100480 2024-03-16T10:57:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f84ff9700 1 -- 172.21.15.153:0/1521931189 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f8001d5e0 con 0x7f5f88100480 2024-03-16T10:57:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f881019a0 con 0x7f5f88100480 2024-03-16T10:57:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.063+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f88101e60 con 0x7f5f88100480 2024-03-16T10:57:47.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.064+0000 7f5f84ff9700 1 -- 172.21.15.153:0/1521931189 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5f80004580 con 0x7f5f88100480 2024-03-16T10:57:47.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.064+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f88063900 con 0x7f5f88100480 2024-03-16T10:57:47.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.064+0000 7f5f84ff9700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5f70038fc0 0x7f5f7003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:47.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.065+0000 7f5f84ff9700 1 -- 172.21.15.153:0/1521931189 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5f8000f050 con 0x7f5f88100480 2024-03-16T10:57:47.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.065+0000 7f5f86ffd700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5f70038fc0 0x7f5f7003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:47.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.065+0000 7f5f86ffd700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5f70038fc0 0x7f5f7003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c006fd0 tx=0x7f5f7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.070+0000 7f5f84ff9700 1 -- 172.21.15.153:0/1521931189 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5f8001c3b0 con 0x7f5f88100480 2024-03-16T10:57:47.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 podman[35461]: 2024-03-16 10:57:47.170527956 +0000 UTC m=+0.533749839 container create c9ce64dc1ae863f5c1aada3bb2704aa5bb45956b25abee34df8f4d7b2f95df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T10:57:47.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:46 smithi078 ceph-mon[32235]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.354+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5f88102090 con 0x7f5f88100480 2024-03-16T10:57:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.355+0000 7f5f84ff9700 1 -- 172.21.15.153:0/1521931189 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f5f80022220 con 0x7f5f88100480 2024-03-16T10:57:47.356 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:57:47.356 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","modified":"2024-03-16T10:56:01.082686Z","created":"2024-03-16T10:56:01.082686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5f70038fc0 msgr2=0x7f5f7003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5f70038fc0 0x7f5f7003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c006fd0 tx=0x7f5f7c006e40 comp rx=0 tx=0).stop 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 msgr2=0x7f5f88101400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88101400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f800076a0 tx=0x7f5f80003ce0 comp rx=0 tx=0).stop 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 shutdown_connections 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5f70038fc0 0x7f5f7003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 --2- 172.21.15.153:0/1521931189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f88100480 0x7f5f88101400 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 >> 172.21.15.153:0/1521931189 conn(0x7f5f880fbe20 msgr2=0x7f5f88103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 shutdown_connections 2024-03-16T10:57:47.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:47.357+0000 7f5f8e090700 1 -- 172.21.15.153:0/1521931189 wait complete. 2024-03-16T10:57:47.359 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-03-16T10:57:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 podman[35461]: 2024-03-16 10:57:47.406395702 +0000 UTC m=+0.769617585 container init c9ce64dc1ae863f5c1aada3bb2704aa5bb45956b25abee34df8f4d7b2f95df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 podman[35461]: 2024-03-16 10:57:47.410027501 +0000 UTC m=+0.773249385 container start c9ce64dc1ae863f5c1aada3bb2704aa5bb45956b25abee34df8f4d7b2f95df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pidfile_write: ignore empty --pid-file 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: load: jerasure load: lrc 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: RocksDB version: 7.9.2 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Git sha 0 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: DB SUMMARY 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: DB Session ID: YOJVUW3QEKFT4NC72A2S 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: CURRENT file: CURRENT 2024-03-16T10:57:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.error_if_exists: 0 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.create_if_missing: 0 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.paranoid_checks: 1 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.env: 0x55ccf0065c60 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.info_log: 0x55ccf0cc3ac0 2024-03-16T10:57:47.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T10:57:47.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.statistics: (nil) 2024-03-16T10:57:47.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.use_fsync: 0 2024-03-16T10:57:47.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_log_file_size: 0 2024-03-16T10:57:47.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T10:57:47.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T10:57:47.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T10:57:47.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.allow_fallocate: 1 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.use_direct_reads: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.db_log_dir: 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.wal_dir: 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T10:57:47.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.write_buffer_manager: 0x55ccf0c961e0 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.unordered_write: 0 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T10:57:47.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.row_cache: None 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.wal_filter: None 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.two_write_queues: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.wal_compression: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.atomic_flush: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T10:57:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.log_readahead_size: 0 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_background_jobs: 2 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_background_compactions: -1 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_subcompactions: 1 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T10:57:47.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_open_files: -1 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_background_flushes: -1 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Compression algorithms supported: 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kZSTD supported: 0 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kXpressCompression supported: 0 2024-03-16T10:57:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kZlibCompression supported: 1 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kSnappyCompression supported: 1 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kLZ4Compression supported: 1 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: kBZip2Compression supported: 0 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.merge_operator: 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_filter: None 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T10:57:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ccf0cc3bc0) 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_type: 0 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: data_block_index_type: 0 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_shortening: 1 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: checksum: 4 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: no_block_cache: 0 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache: 0x55ccf0cc9090 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_options: 2024-03-16T10:57:47.554 INFO:journalctl@ceph.mon.c.smithi153.stdout: capacity : 536870912 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: num_shard_bits : 4 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: strict_capacity_limit : 0 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_compressed: (nil) 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: persistent_cache: (nil) 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_size: 4096 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_size_deviation: 10 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_restart_interval: 16 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_block_restart_interval: 1 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: metadata_block_size: 4096 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: partition_filters: 0 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: use_delta_encoding: 1 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: filter_policy: bloomfilter 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: whole_key_filtering: 1 2024-03-16T10:57:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout: verify_compression: 0 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: format_version: 5 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: enable_index_compression: 1 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_align: 0 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: max_auto_readahead_size: 262144 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: prepopulate_block_cache: 0 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: initial_auto_readahead_size: 8192 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression: NoCompression 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T10:57:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.num_levels: 7 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T10:57:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T10:57:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T10:57:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.table_properties_collectors: 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.inplace_update_support: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.bloom_locality: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.max_successive_merges: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T10:57:47.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.ttl: 2592000 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.enable_blob_files: false 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.min_blob_size: 0 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T10:57:47.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b4c92d3d-425d-443d-8ee2-a20bd22ed1f8 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586667440104, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586667440681, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710586667, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b4c92d3d-425d-443d-8ee2-a20bd22ed1f8", "db_session_id": "YOJVUW3QEKFT4NC72A2S", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586667440786, "job": 1, "event": "recovery_finished"} 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ccf0dbc000 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: DB pointer 0x55ccf0da6000 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** DB Stats ** 2024-03-16T10:57:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-03-16T10:57:47.563 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:57:47.564 INFO:journalctl@ceph.mon.c.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: Block cache BinnedLRUCache@0x55ccf0cc9090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: using public_addr v2:172.21.15.153:0/0 -> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: starting mon.c rank -1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(???) e0 preinit fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).mds e1 new map 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: e1 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: legacy client fscid: -1 2024-03-16T10:57:47.565 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout: No filesystems configured 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:57:47.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Deploying cephadm binary to smithi153 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Added host smithi153 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3870408618' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T10:57:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3870408618' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:47.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c;count:3 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: Deploying daemon mon.c on smithi153 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 ceph-mon[35521]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T10:57:47.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 bash[35461]: c9ce64dc1ae863f5c1aada3bb2704aa5bb45956b25abee34df8f4d7b2f95df5b 2024-03-16T10:57:47.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:47 smithi153 systemd[1]: Started Ceph mon.c for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:57:49.146 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T10:57:49.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T10:57:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:57:52.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:57:52.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: mon.a calling monitor election 2024-03-16T10:57:52.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:52.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:52.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:52.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:52.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: mon.c calling monitor election 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: monmap epoch 2 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: last_changed 2024-03-16T10:57:47.465450+0000 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: min_mon_release 19 (squid) 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: election_strategy: 1 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:57:52.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: fsmap 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: mgrmap e13: x(active, since 50s) 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: overall HEALTH_OK 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:57:52.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: mon.a calling monitor election 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: mon.c calling monitor election 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: monmap epoch 2 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: last_changed 2024-03-16T10:57:47.465450+0000 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: min_mon_release 19 (squid) 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: election_strategy: 1 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: fsmap 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: mgrmap e13: x(active, since 50s) 2024-03-16T10:57:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: overall HEALTH_OK 2024-03-16T10:57:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:57:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:53.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:53 smithi153 ceph-mon[35521]: Deploying daemon mon.b on smithi098 2024-03-16T10:57:53.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 -- 172.21.15.153:0/2431226109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36d80035c0 msgr2=0x7f36d8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 --2- 172.21.15.153:0/2431226109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36d80035c0 0x7f36d8005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f36f0101cd0 tx=0x7f36e0009e80 comp rx=0 tx=0).stop 2024-03-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 -- 172.21.15.153:0/2431226109 shutdown_connections 2024-03-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 --2- 172.21.15.153:0/2431226109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36d80035c0 0x7f36d8005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 -- 172.21.15.153:0/2431226109 >> 172.21.15.153:0/2431226109 conn(0x7f36f00fd870 msgr2=0x7f36f00ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 -- 172.21.15.153:0/2431226109 shutdown_connections 2024-03-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 -- 172.21.15.153:0/2431226109 wait complete. 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.860+0000 7f36f5d13700 1 Processor -- start 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36f5d13700 1 -- start start 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36f5d13700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 0x7f36f019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36f5d13700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36f019db30 0x7f36f01976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36f5d13700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f36f0109ce0 con 0x7f36f019db30 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36f5d13700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36f0109b60 con 0x7f36d80035c0 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36ef7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 0x7f36f019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36ef7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 0x7f36f019d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48196/0 (socket says 172.21.15.153:48196) 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36ef7fe700 1 -- 172.21.15.153:0/601420876 learned_addr learned my addr 172.21.15.153:0/601420876 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36ef7fe700 1 -- 172.21.15.153:0/601420876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 msgr2=0x7f36f019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36ef7fe700 1 -- 172.21.15.153:0/601420876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 msgr2=0x7f36f019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36ef7fe700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 0x7f36f019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36ef7fe700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 0x7f36f019d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.861+0000 7f36eeffd700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36f019db30 0x7f36f01976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36eeffd700 1 -- 172.21.15.153:0/601420876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 msgr2=0x7f36f019d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36eeffd700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 0x7f36f019d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36eeffd700 1 -- 172.21.15.153:0/601420876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f019e0b0 con 0x7f36f019db30 2024-03-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36eeffd700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36f019db30 0x7f36f01976c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36e400b4d0 tx=0x7f36e400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36ecff9700 1 -- 172.21.15.153:0/601420876 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f36e4011630 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36ecff9700 1 -- 172.21.15.153:0/601420876 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36e4011c80 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f0197c60 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36ecff9700 1 -- 172.21.15.153:0/601420876 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f36e4010da0 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.862+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f01981f0 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.863+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f01034e0 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.863+0000 7f36ecff9700 1 -- 172.21.15.153:0/601420876 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f36e4018020 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.863+0000 7f36ecff9700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f36dc038d60 0x7f36dc03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.863+0000 7f36ecff9700 1 -- 172.21.15.153:0/601420876 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f36e404ba90 con 0x7f36f019db30 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.864+0000 7f36ef7fe700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f36dc038d60 0x7f36dc03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.864+0000 7f36ef7fe700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f36dc038d60 0x7f36dc03b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009a30 tx=0x7f36e00090d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:53.869+0000 7f36ecff9700 1 -- 172.21.15.153:0/601420876 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f36e401f020 con 0x7f36f019db30 2024-03-16T10:57:54.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:53 smithi078 ceph-mon[32235]: Deploying daemon mon.b on smithi098 2024-03-16T10:57:54.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:54.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.143+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f36f0198c90 con 0x7f36f019db30 2024-03-16T10:57:54.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.144+0000 7f36ecff9700 1 -- 172.21.15.153:0/601420876 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f36e4018570 con 0x7f36f019db30 2024-03-16T10:57:54.145 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:57:54.145 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","modified":"2024-03-16T10:57:47.465450Z","created":"2024-03-16T10:56:01.082686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T10:57:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f36dc038d60 msgr2=0x7f36dc03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f36dc038d60 0x7f36dc03b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009a30 tx=0x7f36e00090d0 comp rx=0 tx=0).stop 2024-03-16T10:57:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36f019db30 msgr2=0x7f36f01976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36f019db30 0x7f36f01976c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36e400b4d0 tx=0x7f36e400b890 comp rx=0 tx=0).stop 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 shutdown_connections 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f36dc038d60 0x7f36dc03b220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d80035c0 0x7f36f019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 --2- 172.21.15.153:0/601420876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36f019db30 0x7f36f01976c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 >> 172.21.15.153:0/601420876 conn(0x7f36f00fd870 msgr2=0x7f36f00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 shutdown_connections 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:57:54.146+0000 7f36f5d13700 1 -- 172.21.15.153:0/601420876 wait complete. 2024-03-16T10:57:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-03-16T10:57:54.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:54 smithi153 ceph-mon[35521]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:54.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:57:54 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/601420876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:57:55.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:54 smithi078 ceph-mon[32235]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:55.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:57:54 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/601420876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:57:55.675 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T10:57:55.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T10:57:55.921 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:57:56.069 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 podman[35827]: 2024-03-16 10:57:55.903369884 +0000 UTC m=+0.017740062 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 podman[35827]: 2024-03-16 10:57:56.146125925 +0000 UTC m=+0.260496099 container init bfbc5ea10822f23b3c1672f3eeee9f0741e6147062a16e0702f21f479add672f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 podman[35827]: 2024-03-16 10:57:56.149691785 +0000 UTC m=+0.264061955 container start bfbc5ea10822f23b3c1672f3eeee9f0741e6147062a16e0702f21f479add672f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pidfile_write: ignore empty --pid-file 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: load: jerasure load: lrc 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: RocksDB version: 7.9.2 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Git sha 0 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: DB SUMMARY 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: DB Session ID: 42I7JVCGBPL7Z8NBWUDE 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: CURRENT file: CURRENT 2024-03-16T10:57:56.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.error_if_exists: 0 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.create_if_missing: 0 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.paranoid_checks: 1 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.env: 0x5647b715ac60 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.info_log: 0x5647b8459ac0 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.statistics: (nil) 2024-03-16T10:57:56.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.use_fsync: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_log_file_size: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.allow_fallocate: 1 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.use_direct_reads: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.db_log_dir: 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.wal_dir: 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T10:57:56.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.write_buffer_manager: 0x5647b842c1e0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.unordered_write: 0 2024-03-16T10:57:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T10:57:56.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T10:57:56.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T10:57:56.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T10:57:56.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.row_cache: None 2024-03-16T10:57:56.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.wal_filter: None 2024-03-16T10:57:56.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T10:57:56.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T10:57:56.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.two_write_queues: 0 2024-03-16T10:57:56.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.wal_compression: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.atomic_flush: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.log_readahead_size: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_background_jobs: 2 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_background_compactions: -1 2024-03-16T10:57:56.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_subcompactions: 1 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_open_files: -1 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_background_flushes: -1 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Compression algorithms supported: 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T10:57:56.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kZSTD supported: 0 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kXpressCompression supported: 0 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kZlibCompression supported: 1 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kSnappyCompression supported: 1 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kLZ4Compression supported: 1 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: kBZip2Compression supported: 0 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.merge_operator: 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_filter: None 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T10:57:56.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5647b8459bc0) 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_type: 0 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: data_block_index_type: 0 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_shortening: 1 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: checksum: 4 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: no_block_cache: 0 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache: 0x5647b845f090 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-03-16T10:57:56.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_options: 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: capacity : 536870912 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: num_shard_bits : 4 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: strict_capacity_limit : 0 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_compressed: (nil) 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: persistent_cache: (nil) 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_size: 4096 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_size_deviation: 10 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_restart_interval: 16 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_block_restart_interval: 1 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: metadata_block_size: 4096 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: partition_filters: 0 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: use_delta_encoding: 1 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: filter_policy: bloomfilter 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: whole_key_filtering: 1 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: verify_compression: 0 2024-03-16T10:57:56.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout: format_version: 5 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout: enable_index_compression: 1 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_align: 0 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout: max_auto_readahead_size: 262144 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout: prepopulate_block_cache: 0 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout: initial_auto_readahead_size: 8192 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression: NoCompression 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.num_levels: 7 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T10:57:56.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:57:56.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T10:57:56.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T10:57:56.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.table_properties_collectors: 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.inplace_update_support: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.bloom_locality: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.max_successive_merges: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.ttl: 2592000 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T10:57:56.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.enable_blob_files: false 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.min_blob_size: 0 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ffd421e9-da93-4d7e-ae6e-b169682fca09 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586676179891, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T10:57:56.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586676180403, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710586676, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ffd421e9-da93-4d7e-ae6e-b169682fca09", "db_session_id": "42I7JVCGBPL7Z8NBWUDE", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710586676180500, "job": 1, "event": "recovery_finished"} 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5647b8552000 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: DB pointer 0x5647b853c000 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** DB Stats ** 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:57:56.346 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:57:56.347 INFO:journalctl@ceph.mon.b.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: Block cache BinnedLRUCache@0x5647b845f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: using public_addr v2:172.21.15.98:0/0 -> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: starting mon.b rank -1 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(???) e0 preinit fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:56.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).mds e1 new map 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout: e1 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout: legacy client fscid: -1 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout: No filesystems configured 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:57:56.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:56.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Deploying cephadm binary to smithi153 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Added host smithi153 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3870408618' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:57:56.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3870408618' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c;count:3 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.353 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Deploying daemon mon.c on smithi153 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.a calling monitor election 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.c calling monitor election 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.354 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: monmap epoch 2 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: last_changed 2024-03-16T10:57:47.465450+0000 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: min_mon_release 19 (squid) 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: election_strategy: 1 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: fsmap 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mgrmap e13: x(active, since 50s) 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: overall HEALTH_OK 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: Deploying daemon mon.b on smithi098 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/601420876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 ceph-mon[35848]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 bash[35827]: bfbc5ea10822f23b3c1672f3eeee9f0741e6147062a16e0702f21f479add672f 2024-03-16T10:57:56.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:57:56 smithi098 systemd[1]: Started Ceph mon.b for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:58:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: mon.c calling monitor election 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: mon.a calling monitor election 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: monmap epoch 3 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:01.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: last_changed 2024-03-16T10:57:56.207688+0000 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: min_mon_release 19 (squid) 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: election_strategy: 1 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: fsmap 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: mgrmap e13: x(active, since 59s) 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.710+0000 7f186b670700 1 -- 172.21.15.153:0/2737125540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 msgr2=0x7f184c005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:01.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.710+0000 7f186b670700 1 --2- 172.21.15.153:0/2737125540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 0x7f184c005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f185c005fa0 tx=0x7f185c00ff70 comp rx=0 tx=0).stop 2024-03-16T10:58:01.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.710+0000 7f186b670700 1 -- 172.21.15.153:0/2737125540 shutdown_connections 2024-03-16T10:58:01.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.710+0000 7f186b670700 1 --2- 172.21.15.153:0/2737125540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 0x7f184c005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:01.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.710+0000 7f186b670700 1 -- 172.21.15.153:0/2737125540 >> 172.21.15.153:0/2737125540 conn(0x7f18640fc770 msgr2=0x7f18640feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:01.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.710+0000 7f186b670700 1 -- 172.21.15.153:0/2737125540 shutdown_connections 2024-03-16T10:58:01.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.710+0000 7f186b670700 1 -- 172.21.15.153:0/2737125540 wait complete. 2024-03-16T10:58:01.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 Processor -- start 2024-03-16T10:58:01.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 -- start start 2024-03-16T10:58:01.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:58:01.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: mon.c calling monitor election 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: mon.a calling monitor election 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: monmap epoch 3 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: last_changed 2024-03-16T10:57:56.207688+0000 2024-03-16T10:58:01.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: min_mon_release 19 (squid) 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: election_strategy: 1 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: fsmap 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: mgrmap e13: x(active, since 59s) 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.714 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.715 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 0x7f1864100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 0x7f1864101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 0x7f18641a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1864109c90 con 0x7f1864103020 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1864109b10 con 0x7f18641a3a60 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186b670700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1864109e10 con 0x7f18641011c0 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1868c0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 0x7f1864101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1868c0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 0x7f1864101640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40480/0 (socket says 172.21.15.153:40480) 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1868c0b700 1 -- 172.21.15.153:0/442112072 learned_addr learned my addr 172.21.15.153:0/442112072 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f186940c700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 0x7f1864100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1868c0b700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 msgr2=0x7f1864101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1868c0b700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 msgr2=0x7f1864101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1868c0b700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 0x7f1864101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1869c0d700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 0x7f18641a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.711+0000 7f1868c0b700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 0x7f1864101640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f1869c0d700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 msgr2=0x7f18641a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f1869c0d700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 msgr2=0x7f18641a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f1869c0d700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 0x7f18641a5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f1869c0d700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 0x7f18641a5e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186940c700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 msgr2=0x7f1864101640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186940c700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 0x7f1864101640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186940c700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 msgr2=0x7f18641a5e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186940c700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 0x7f18641a5e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:01.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186940c700 1 -- 172.21.15.153:0/442112072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18641a6580 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186940c700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 0x7f1864100c80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f185c01f000 tx=0x7f185c01d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f18627fc700 1 -- 172.21.15.153:0/442112072 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c02b020 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f18627fc700 1 -- 172.21.15.153:0/442112072 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f185c01dd40 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f18627fc700 1 -- 172.21.15.153:0/442112072 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c022040 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18641a6810 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.712+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18641a6d50 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.713+0000 7f18627fc700 1 -- 172.21.15.153:0/442112072 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f185c02e020 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.713+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1864063900 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.713+0000 7f18627fc700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1850039450 0x7f185003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.713+0000 7f18627fc700 1 -- 172.21.15.153:0/442112072 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f185c04ec70 con 0x7f1864103020 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.714+0000 7f1868c0b700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1850039450 0x7f185003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.714+0000 7f1868c0b700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1850039450 0x7f185003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1858009180 tx=0x7f1858008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:01.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:01.719+0000 7f18627fc700 1 -- 172.21.15.153:0/442112072 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f185c035020 con 0x7f1864103020 2024-03-16T10:58:02.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.020+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f18641a7080 con 0x7f1864103020 2024-03-16T10:58:02.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.021+0000 7f18627fc700 1 -- 172.21.15.153:0/442112072 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f185c010c20 con 0x7f1864103020 2024-03-16T10:58:02.022 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T10:58:02.023 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":3,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","modified":"2024-03-16T10:57:56.207688Z","created":"2024-03-16T10:56:01.082686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1850039450 msgr2=0x7f185003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1850039450 0x7f185003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1858009180 tx=0x7f1858008040 comp rx=0 tx=0).stop 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 msgr2=0x7f1864100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 0x7f1864100c80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f185c01f000 tx=0x7f185c01d560 comp rx=0 tx=0).stop 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 shutdown_connections 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1850039450 0x7f185003b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18641a3a60 0x7f18641a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1864103020 0x7f1864100c80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 --2- 172.21.15.153:0/442112072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641011c0 0x7f1864101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:02.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 >> 172.21.15.153:0/442112072 conn(0x7f18640fc770 msgr2=0x7f18640fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:02.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 shutdown_connections 2024-03-16T10:58:02.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:02.024+0000 7f186b670700 1 -- 172.21.15.153:0/442112072 wait complete. 2024-03-16T10:58:02.026 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 3 2024-03-16T10:58:02.626 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:02.626 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:02 smithi153 ceph-mon[35521]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:02.626 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:02.626 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:02 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/442112072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:58:02.626 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:02.651 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-16T10:58:02.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-16T10:58:02.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:02.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:02 smithi078 ceph-mon[32235]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:02.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:02.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:02 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/442112072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:58:02.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:02.898 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.c calling monitor election 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.a calling monitor election 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:03.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: monmap epoch 3 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: last_changed 2024-03-16T10:57:56.207688+0000 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: min_mon_release 19 (squid) 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: election_strategy: 1 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:58:03.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: fsmap 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mgrmap e13: x(active, since 59s) 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:03.650 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/442112072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.b calling monitor election 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.b calling monitor election 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.a calling monitor election 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.c calling monitor election 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: monmap epoch 3 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: last_changed 2024-03-16T10:57:56.207688+0000 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: min_mon_release 19 (squid) 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: election_strategy: 1 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:58:03.651 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: fsmap 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: mgrmap e13: x(active, since 61s) 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: Cluster is now healthy 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: overall HEALTH_OK 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:03.652 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: mon.b calling monitor election 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: mon.b calling monitor election 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: mon.a calling monitor election 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: mon.c calling monitor election 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: monmap epoch 3 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: last_changed 2024-03-16T10:57:56.207688+0000 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: min_mon_release 19 (squid) 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: election_strategy: 1 2024-03-16T10:58:04.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: fsmap 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: mgrmap e13: x(active, since 61s) 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: Cluster is now healthy 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: overall HEALTH_OK 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:04.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: mon.b calling monitor election 2024-03-16T10:58:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: mon.b calling monitor election 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: mon.a calling monitor election 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: mon.c calling monitor election 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: monmap epoch 3 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: last_changed 2024-03-16T10:57:56.207688+0000 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: created 2024-03-16T10:56:01.082686+0000 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: min_mon_release 19 (squid) 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: election_strategy: 1 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: 2: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: fsmap 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:58:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: mgrmap e13: x(active, since 61s) 2024-03-16T10:58:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T10:58:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: Cluster is now healthy 2024-03-16T10:58:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: overall HEALTH_OK 2024-03-16T10:58:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.562+0000 7f7a6780b700 1 -- 172.21.15.78:0/1021052975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 msgr2=0x7f7a601011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.562+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1021052975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a601011f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7a50009910 tx=0x7f7a50009c20 comp rx=0 tx=0).stop 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.562+0000 7f7a6780b700 1 -- 172.21.15.78:0/1021052975 shutdown_connections 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.562+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1021052975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a601011f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.562+0000 7f7a6780b700 1 -- 172.21.15.78:0/1021052975 >> 172.21.15.78:0/1021052975 conn(0x7f7a600fc6c0 msgr2=0x7f7a600feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 -- 172.21.15.78:0/1021052975 shutdown_connections 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 -- 172.21.15.78:0/1021052975 wait complete. 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 Processor -- start 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 -- start start 2024-03-16T10:58:04.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a6019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6019dbc0 0x7f7a60197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a60197c90 0x7f7a60198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6010b090 con 0x7f7a60100e10 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6010af10 con 0x7f7a6019dbc0 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a6780b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6010b210 con 0x7f7a60197c90 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a655a7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a6019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a655a7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a6019d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47862/0 (socket says 172.21.15.78:47862) 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.563+0000 7f7a655a7700 1 -- 172.21.15.78:0/1330237860 learned_addr learned my addr 172.21.15.78:0/1330237860 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a64da6700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6019dbc0 0x7f7a60197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a65da8700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a60197c90 0x7f7a60198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a655a7700 1 -- 172.21.15.78:0/1330237860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a60197c90 msgr2=0x7f7a60198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a655a7700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a60197c90 0x7f7a60198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a655a7700 1 -- 172.21.15.78:0/1330237860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6019dbc0 msgr2=0x7f7a60197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a655a7700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6019dbc0 0x7f7a60197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:04.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a655a7700 1 -- 172.21.15.78:0/1330237860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a601a4430 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a655a7700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a6019d680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7a50005b40 tx=0x7f7a50005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a567fc700 1 -- 172.21.15.78:0/1330237860 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50026070 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a567fc700 1 -- 172.21.15.78:0/1330237860 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a5000b6f0 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a601a46c0 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a567fc700 1 -- 172.21.15.78:0/1330237860 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a500038e0 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.564+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a601a4c50 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.565+0000 7f7a567fc700 1 -- 172.21.15.78:0/1330237860 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7a50029070 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.565+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a601022f0 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.565+0000 7f7a567fc700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7a4c039450 0x7f7a4c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.565+0000 7f7a567fc700 1 -- 172.21.15.78:0/1330237860 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7a5004cc30 con 0x7f7a60100e10 2024-03-16T10:58:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.565+0000 7f7a64da6700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7a4c039450 0x7f7a4c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:04.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.566+0000 7f7a64da6700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7a4c039450 0x7f7a4c03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009850 tx=0x7f7a5c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:04.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.570+0000 7f7a567fc700 1 -- 172.21.15.78:0/1330237860 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a50013070 con 0x7f7a60100e10 2024-03-16T10:58:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.780+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f7a6002db70 con 0x7f7a60100e10 2024-03-16T10:58:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.780+0000 7f7a567fc700 1 -- 172.21.15.78:0/1330237860 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f7a50030090 con 0x7f7a60100e10 2024-03-16T10:58:04.781 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:04.781 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-03-16T10:58:04.781 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T10:58:04.782 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-16T10:58:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7a4c039450 msgr2=0x7f7a4c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7a4c039450 0x7f7a4c03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009850 tx=0x7f7a5c008040 comp rx=0 tx=0).stop 2024-03-16T10:58:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 msgr2=0x7f7a6019d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a6019d680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7a50005b40 tx=0x7f7a50005030 comp rx=0 tx=0).stop 2024-03-16T10:58:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 shutdown_connections 2024-03-16T10:58:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7a4c039450 0x7f7a4c03b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a60197c90 0x7f7a60198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a60100e10 0x7f7a6019d680 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 --2- 172.21.15.78:0/1330237860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6019dbc0 0x7f7a60197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 >> 172.21.15.78:0/1330237860 conn(0x7f7a600fc6c0 msgr2=0x7f7a601049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 shutdown_connections 2024-03-16T10:58:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:04.782+0000 7f7a6780b700 1 -- 172.21.15.78:0/1330237860 wait complete. 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Updating smithi078:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: Reconfiguring daemon mon.a on smithi078 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:05.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:04 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1330237860' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Updating smithi078:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: Reconfiguring daemon mon.a on smithi078 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:05.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:04 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1330237860' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:05.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Updating smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Updating smithi078:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Updating smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: Reconfiguring daemon mon.a on smithi078 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:58:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:04 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1330237860' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:05.404 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-16T10:58:05.404 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:58:05.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T10:58:05.473 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:58:05.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:58:05.543 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:58:05.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T10:58:05.577 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:58:05.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:58:05.647 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:58:05.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T10:58:05.681 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:58:05.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:58:05.752 INFO:tasks.cephadm:Adding mgr.x on smithi078 2024-03-16T10:58:05.753 INFO:tasks.cephadm:Adding mgr.y on smithi098 2024-03-16T10:58:05.753 INFO:tasks.cephadm:Adding mgr.z on smithi153 2024-03-16T10:58:05.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi078=x;smithi098=y;smithi153=z' 2024-03-16T10:58:06.026 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:58:07.061 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:06 smithi098 ceph-mon[35848]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:07.061 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:07.061 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:07.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:07.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:07.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:07.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:06 smithi153 ceph-mon[35521]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:07.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:07.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:07.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:07.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:07.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:06 smithi078 ceph-mon[32235]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:07.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.567+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/3447002997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88103480 msgr2=0x7f5e881038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:07.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.567+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/3447002997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88103480 0x7f5e881038a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009910 tx=0x7f5e78009c20 comp rx=0 tx=0).stop 2024-03-16T10:58:07.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.567+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/3447002997 shutdown_connections 2024-03-16T10:58:07.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.567+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/3447002997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88103480 0x7f5e881038a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.567+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/3447002997 >> 172.21.15.153:0/3447002997 conn(0x7f5e880fd3c0 msgr2=0x7f5e880ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:07.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.567+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/3447002997 shutdown_connections 2024-03-16T10:58:07.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.567+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/3447002997 wait complete. 2024-03-16T10:58:07.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 Processor -- start 2024-03-16T10:58:07.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 -- start start 2024-03-16T10:58:07.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e88103480 0x7f5e88076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:07.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e880746c0 0x7f5e88074b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:07.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88075080 0x7f5e881a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:07.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5e88107e00 con 0x7f5e88075080 2024-03-16T10:58:07.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e88107c80 con 0x7f5e880746c0 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8cd6c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5e88107f80 con 0x7f5e88103480 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e88103480 0x7f5e88076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e88103480 0x7f5e88076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35602/0 (socket says 172.21.15.153:35602) 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 -- 172.21.15.153:0/1908284443 learned_addr learned my addr 172.21.15.153:0/1908284443 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e86d9d700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88075080 0x7f5e881a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 -- 172.21.15.153:0/1908284443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e880746c0 msgr2=0x7f5e88074b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e880746c0 0x7f5e88074b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 -- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88075080 msgr2=0x7f5e881a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88075080 0x7f5e881a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 -- 172.21.15.153:0/1908284443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e881a1cb0 con 0x7f5e88103480 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.568+0000 7f5e8659c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e88103480 0x7f5e88076070 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5e78005e00 tx=0x7f5e78005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.569+0000 7f5e777fe700 1 -- 172.21.15.153:0/1908284443 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78026070 con 0x7f5e88103480 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.569+0000 7f5e777fe700 1 -- 172.21.15.153:0/1908284443 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e7800be10 con 0x7f5e88103480 2024-03-16T10:58:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.569+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e881a1f40 con 0x7f5e88103480 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.569+0000 7f5e777fe700 1 -- 172.21.15.153:0/1908284443 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78017440 con 0x7f5e88103480 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.569+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e881a2410 con 0x7f5e88103480 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.570+0000 7f5e777fe700 1 -- 172.21.15.153:0/1908284443 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5e78029070 con 0x7f5e88103480 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.570+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e88063900 con 0x7f5e88103480 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.570+0000 7f5e777fe700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5e70039450 0x7f5e7003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.570+0000 7f5e777fe700 1 -- 172.21.15.153:0/1908284443 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5e7804cf30 con 0x7f5e88103480 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.570+0000 7f5e85d9b700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5e70039450 0x7f5e7003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:07.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.570+0000 7f5e85d9b700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5e70039450 0x7f5e7003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0098e0 tx=0x7f5e7c006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:07.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.575+0000 7f5e777fe700 1 -- 172.21.15.153:0/1908284443 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e78013070 con 0x7f5e88103480 2024-03-16T10:58:07.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.784+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi098=y;smithi153=z", "target": ["mon-mgr", ""]}) -- 0x7f5e88062710 con 0x7f5e70039450 2024-03-16T10:58:07.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.791+0000 7f5e777fe700 1 -- 172.21.15.153:0/1908284443 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5e88062710 con 0x7f5e70039450 2024-03-16T10:58:07.792 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mgr update... 2024-03-16T10:58:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5e70039450 msgr2=0x7f5e7003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5e70039450 0x7f5e7003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0098e0 tx=0x7f5e7c006dd0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e88103480 msgr2=0x7f5e88076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e88103480 0x7f5e88076070 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5e78005e00 tx=0x7f5e78005050 comp rx=0 tx=0).stop 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 shutdown_connections 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5e70039450 0x7f5e7003b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e88103480 0x7f5e88076070 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e88075080 0x7f5e881a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 --2- 172.21.15.153:0/1908284443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e880746c0 0x7f5e88074b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 >> 172.21.15.153:0/1908284443 conn(0x7f5e880fd3c0 msgr2=0x7f5e880fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 shutdown_connections 2024-03-16T10:58:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:58:07.793+0000 7f5e8cd6c700 1 -- 172.21.15.153:0/1908284443 wait complete. 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: Reconfiguring mon.b (monmap changed)... 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: Reconfiguring daemon mon.b on smithi098 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:08.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:07 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: Reconfiguring mon.b (monmap changed)... 2024-03-16T10:58:08.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: Reconfiguring daemon mon.b on smithi098 2024-03-16T10:58:08.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:08.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:08.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:08.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:07 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: Reconfiguring mon.b (monmap changed)... 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: Reconfiguring daemon mon.b on smithi098 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:07 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:08.384 DEBUG:teuthology.orchestra.run.smithi098:mgr.y> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.y.service 2024-03-16T10:58:08.387 DEBUG:teuthology.orchestra.run.smithi153:mgr.z> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.z.service 2024-03-16T10:58:08.389 INFO:tasks.cephadm:Deploying OSDs... 2024-03-16T10:58:08.389 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T10:58:08.389 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T10:58:08.409 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T10:58:08.409 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-16 10:57:14.734661556 +0000 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-16 10:52:48.079662312 +0000 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-16 10:52:48.079662312 +0000 2024-03-16T10:58:08.468 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-16 10:52:48.079662312 +0000 2024-03-16T10:58:08.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T10:58:08.539 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-16T10:58:08.539 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-16T10:58:08.539 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000311598 s, 1.6 MB/s 2024-03-16T10:58:08.541 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T10:58:08.598 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-16 10:57:14.760661124 +0000 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-16 10:52:48.423656488 +0000 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-16 10:52:48.423656488 +0000 2024-03-16T10:58:08.655 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-16 10:52:48.423656488 +0000 2024-03-16T10:58:08.655 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T10:58:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-16T10:58:08.722 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-16T10:58:08.722 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.00018853 s, 2.7 MB/s 2024-03-16T10:58:08.723 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T10:58:08.779 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-03-16T10:58:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T10:58:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:08.836 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-16T10:58:08.836 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:08.836 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:08.837 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-16 10:57:14.785660709 +0000 2024-03-16T10:58:08.837 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-16 10:52:48.761650766 +0000 2024-03-16T10:58:08.837 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-16 10:52:48.761650766 +0000 2024-03-16T10:58:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-16 10:52:48.761650766 +0000 2024-03-16T10:58:08.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T10:58:08.901 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-16T10:58:08.901 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-16T10:58:08.901 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000176807 s, 2.9 MB/s 2024-03-16T10:58:08.902 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T10:58:08.958 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-03-16T10:58:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T10:58:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:09.013 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-16T10:58:09.013 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:09.013 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:09.013 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-16 10:57:14.810660293 +0000 2024-03-16T10:58:09.014 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-16 10:52:49.085645280 +0000 2024-03-16T10:58:09.014 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-16 10:52:49.085645280 +0000 2024-03-16T10:58:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-16 10:52:49.085645280 +0000 2024-03-16T10:58:09.014 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T10:58:09.079 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-16T10:58:09.079 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-16T10:58:09.079 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000219094 s, 2.3 MB/s 2024-03-16T10:58:09.080 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T10:58:09.136 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T10:58:09.136 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T10:58:09.154 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T10:58:09.154 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-16T10:58:09.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:08 smithi153 ceph-mon[35521]: Reconfiguring mon.c (monmap changed)... 2024-03-16T10:58:09.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:08 smithi153 ceph-mon[35521]: Reconfiguring daemon mon.c on smithi153 2024-03-16T10:58:09.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:08 smithi153 ceph-mon[35521]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi098=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:09.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:08 smithi153 ceph-mon[35521]: Saving service mgr spec with placement smithi078=x;smithi098=y;smithi153=z;count:3 2024-03-16T10:58:09.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:08 smithi153 ceph-mon[35521]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:09.210 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T10:58:09.210 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:09.210 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-16T10:58:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 10:57:55.707291018 +0000 2024-03-16T10:58:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 10:52:48.991471802 +0000 2024-03-16T10:58:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 10:52:48.991471802 +0000 2024-03-16T10:58:09.211 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 10:52:48.991471802 +0000 2024-03-16T10:58:09.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T10:58:09.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:08 smithi098 ceph-mon[35848]: Reconfiguring mon.c (monmap changed)... 2024-03-16T10:58:09.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:08 smithi098 ceph-mon[35848]: Reconfiguring daemon mon.c on smithi153 2024-03-16T10:58:09.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:08 smithi098 ceph-mon[35848]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi098=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:09.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:08 smithi098 ceph-mon[35848]: Saving service mgr spec with placement smithi078=x;smithi098=y;smithi153=z;count:3 2024-03-16T10:58:09.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:08 smithi098 ceph-mon[35848]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T10:58:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T10:58:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000283086 s, 1.8 MB/s 2024-03-16T10:58:09.243 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T10:58:09.300 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-16T10:58:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:08 smithi078 ceph-mon[32235]: Reconfiguring mon.c (monmap changed)... 2024-03-16T10:58:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:08 smithi078 ceph-mon[32235]: Reconfiguring daemon mon.c on smithi153 2024-03-16T10:58:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:08 smithi078 ceph-mon[32235]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi098=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:08 smithi078 ceph-mon[32235]: Saving service mgr spec with placement smithi078=x;smithi098=y;smithi153=z;count:3 2024-03-16T10:58:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:08 smithi078 ceph-mon[32235]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T10:58:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:09.356 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-16T10:58:09.357 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:09.357 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:09.357 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 10:57:55.707291018 +0000 2024-03-16T10:58:09.357 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 10:52:49.329467424 +0000 2024-03-16T10:58:09.357 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 10:52:49.329467424 +0000 2024-03-16T10:58:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 10:52:49.329467424 +0000 2024-03-16T10:58:09.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T10:58:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T10:58:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T10:58:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000182367 s, 2.8 MB/s 2024-03-16T10:58:09.425 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T10:58:09.481 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-16T10:58:09.536 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T10:58:09.536 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:09.536 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-16T10:58:09.537 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:09.537 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:09.537 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 10:57:55.707291018 +0000 2024-03-16T10:58:09.537 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 10:52:49.669463019 +0000 2024-03-16T10:58:09.537 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 10:52:49.669463019 +0000 2024-03-16T10:58:09.537 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 10:52:49.669463019 +0000 2024-03-16T10:58:09.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T10:58:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T10:58:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T10:58:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000191334 s, 2.7 MB/s 2024-03-16T10:58:09.603 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T10:58:09.659 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-16T10:58:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T10:58:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:09.714 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-16T10:58:09.714 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:09.714 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:09.714 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 10:57:55.708291005 +0000 2024-03-16T10:58:09.714 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 10:52:49.993458822 +0000 2024-03-16T10:58:09.714 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 10:52:49.993458822 +0000 2024-03-16T10:58:09.714 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 10:52:49.993458822 +0000 2024-03-16T10:58:09.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T10:58:09.778 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T10:58:09.778 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T10:58:09.778 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000169239 s, 3.0 MB/s 2024-03-16T10:58:09.779 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T10:58:09.835 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T10:58:09.835 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T10:58:09.851 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T10:58:09.851 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 10:57:46.324968408 +0000 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 10:52:48.816371231 +0000 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 10:52:48.816371231 +0000 2024-03-16T10:58:09.911 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 10:52:48.816371231 +0000 2024-03-16T10:58:09.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T10:58:09.978 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T10:58:09.978 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T10:58:09.978 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000276028 s, 1.9 MB/s 2024-03-16T10:58:09.980 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T10:58:10.038 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-16T10:58:10.092 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 10:57:46.324968408 +0000 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 10:52:49.146365375 +0000 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 10:52:49.146365375 +0000 2024-03-16T10:58:10.093 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 10:52:49.146365375 +0000 2024-03-16T10:58:10.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T10:58:10.158 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T10:58:10.159 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T10:58:10.159 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000179361 s, 2.9 MB/s 2024-03-16T10:58:10.160 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T10:58:10.215 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-16T10:58:10.270 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 10:57:46.325968390 +0000 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 10:52:49.502359057 +0000 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 10:52:49.502359057 +0000 2024-03-16T10:58:10.271 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 10:52:49.502359057 +0000 2024-03-16T10:58:10.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T10:58:10.336 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T10:58:10.404 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T10:58:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000200992 s, 2.5 MB/s 2024-03-16T10:58:10.406 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T10:58:10.462 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-16T10:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T10:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-16T10:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 10:57:46.325968390 +0000 2024-03-16T10:58:10.518 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 10:52:49.826353308 +0000 2024-03-16T10:58:10.519 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 10:52:49.826353308 +0000 2024-03-16T10:58:10.519 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 10:52:49.826353308 +0000 2024-03-16T10:58:10.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T10:58:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T10:58:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T10:58:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000201356 s, 2.5 MB/s 2024-03-16T10:58:10.585 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:10.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:10.608 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:10 smithi153 ceph-mon[35521]: Deploying daemon mgr.z on smithi153 2024-03-16T10:58:10.622 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2024-03-16T10:58:10.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:10.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:58:10.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:10.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:10.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:10 smithi078 ceph-mon[32235]: Deploying daemon mgr.z on smithi153 2024-03-16T10:58:10.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:10.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:10 smithi098 ceph-mon[35848]: Deploying daemon mgr.z on smithi153 2024-03-16T10:58:10.856 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:58:12.043 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:11 smithi153 systemd[1]: Starting Ceph mgr.z for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T10:58:12.355 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 podman[37359]: 2024-03-16 10:58:11.999181036 +0000 UTC m=+0.017249257 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:58:12.355 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 podman[37359]: 2024-03-16 10:58:12.116870748 +0000 UTC m=+0.134938953 container create 50074f2f014f5f19a067fa0d348f1f1ee9dfe5b318183c83c414749412e9555e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T10:58:12.355 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 podman[37359]: 2024-03-16 10:58:12.327447367 +0000 UTC m=+0.345515575 container init 50074f2f014f5f19a067fa0d348f1f1ee9dfe5b318183c83c414749412e9555e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-16T10:58:12.355 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 podman[37359]: 2024-03-16 10:58:12.330426933 +0000 UTC m=+0.348495137 container start 50074f2f014f5f19a067fa0d348f1f1ee9dfe5b318183c83c414749412e9555e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T10:58:12.355 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mgr[37378]: -- 172.21.15.153:0/2148544125 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eaf17285a0 con 0x55eaf16fd000 2024-03-16T10:58:12.793 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 bash[37359]: 50074f2f014f5f19a067fa0d348f1f1ee9dfe5b318183c83c414749412e9555e 2024-03-16T10:58:12.793 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 systemd[1]: Started Ceph mgr.z for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:58:12.830 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T10:58:12.830 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T10:58:12.830 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-16T10:58:12.830 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-16T10:58:12.830 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114687 s, 914 MB/s 2024-03-16T10:58:12.830 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-16T10:58:13.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:13.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:13.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:58:13.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:13.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:13.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:13.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:13.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:58:13.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:13.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:58:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:13.294 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:12.880+0000 7f62d541a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:58:13.294 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:12 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:12.974+0000 7f62d541a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:58:13.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2024-03-16T10:58:13.748 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:58:13.751 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:13 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:13.619+0000 7f62d541a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:58:13.752 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:13 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:13.713+0000 7f62d541a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:58:14.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:13 smithi153 ceph-mon[35521]: Deploying daemon mgr.y on smithi098 2024-03-16T10:58:14.044 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:13 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:13.861+0000 7f62d541a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:58:14.044 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:13 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:13.961+0000 7f62d541a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:58:14.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:13 smithi098 ceph-mon[35848]: Deploying daemon mgr.y on smithi098 2024-03-16T10:58:14.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:13 smithi078 ceph-mon[32235]: Deploying daemon mgr.y on smithi098 2024-03-16T10:58:14.318 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:14.047+0000 7f62d541a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:58:14.318 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:14.317+0000 7f62d541a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:58:14.610 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:14.427+0000 7f62d541a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:58:14.611 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:14.514+0000 7f62d541a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:58:14.611 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:14.609+0000 7f62d541a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:58:14.688 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 systemd[1]: Starting Ceph mgr.y for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T10:58:14.955 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:14.789+0000 7f62d541a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:58:14.955 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 podman[37236]: 2024-03-16 10:58:14.591514961 +0000 UTC m=+0.018114372 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:58:14.956 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 podman[37236]: 2024-03-16 10:58:14.707336698 +0000 UTC m=+0.133936100 container create eef3db8b636f6c01f04a9d856bd79bbfbf7003abdb93323b5a22a74e29183d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:58:14.956 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 podman[37236]: 2024-03-16 10:58:14.909987869 +0000 UTC m=+0.336587282 container init eef3db8b636f6c01f04a9d856bd79bbfbf7003abdb93323b5a22a74e29183d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-16T10:58:14.956 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 podman[37236]: 2024-03-16 10:58:14.91249142 +0000 UTC m=+0.339090822 container start eef3db8b636f6c01f04a9d856bd79bbfbf7003abdb93323b5a22a74e29183d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T10:58:14.956 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 ceph-mgr[37255]: -- 172.21.15.98:0/1332573142 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55955dfd25a0 con 0x55955dfa6c00 2024-03-16T10:58:15.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:14 smithi098 ceph-mon[35848]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:15.215 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 bash[37236]: eef3db8b636f6c01f04a9d856bd79bbfbf7003abdb93323b5a22a74e29183d43 2024-03-16T10:58:15.215 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:14 smithi098 systemd[1]: Started Ceph mgr.y for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T10:58:15.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:14 smithi078 ceph-mon[32235]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:15.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:14 smithi153 ceph-mon[35521]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 -- 172.21.15.78:0/355817657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103520 msgr2=0x7f60c8103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 --2- 172.21.15.78:0/355817657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103520 0x7f60c8103980 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f60b80099c0 tx=0x7f60b8009cd0 comp rx=0 tx=0).stop 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 -- 172.21.15.78:0/355817657 shutdown_connections 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 --2- 172.21.15.78:0/355817657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c8102320 0x7f60c8102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 --2- 172.21.15.78:0/355817657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103ec0 0x7f60c810a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 --2- 172.21.15.78:0/355817657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103520 0x7f60c8103980 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 -- 172.21.15.78:0/355817657 >> 172.21.15.78:0/355817657 conn(0x7f60c80fd8e0 msgr2=0x7f60c80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 -- 172.21.15.78:0/355817657 shutdown_connections 2024-03-16T10:58:15.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.344+0000 7f60cfad5700 1 -- 172.21.15.78:0/355817657 wait complete. 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 Processor -- start 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 -- start start 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c8102320 0x7f60c8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103520 0x7f60c81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103ec0 0x7f60c819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810ddc0 con 0x7f60c8103ec0 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810dc40 con 0x7f60c8103520 2024-03-16T10:58:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cfad5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810df40 con 0x7f60c8102320 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cd070700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103520 0x7f60c81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60ce072700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103ec0 0x7f60c819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cd070700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103520 0x7f60c81989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:40394/0 (socket says 172.21.15.78:40394) 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cd070700 1 -- 172.21.15.78:0/3970603162 learned_addr learned my addr 172.21.15.78:0/3970603162 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cd871700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c8102320 0x7f60c8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60ce072700 1 -- 172.21.15.78:0/3970603162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c8102320 msgr2=0x7f60c8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60ce072700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c8102320 0x7f60c8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60ce072700 1 -- 172.21.15.78:0/3970603162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103520 msgr2=0x7f60c81989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60ce072700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103520 0x7f60c81989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60ce072700 1 -- 172.21.15.78:0/3970603162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c81a00c0 con 0x7f60c8103ec0 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60cd070700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103520 0x7f60c81989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.345+0000 7f60ce072700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103ec0 0x7f60c819fa20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f60c0007f40 tx=0x7f60c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.346+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0004db0 con 0x7f60c8103ec0 2024-03-16T10:58:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.346+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c81a03b0 con 0x7f60c8103ec0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.346+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c0009e20 con 0x7f60c8103ec0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.346+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0015400 con 0x7f60c8103ec0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.346+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c81a08a0 con 0x7f60c8103ec0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.346+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f60c0005240 con 0x7f60c8103ec0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.347+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c8068f90 con 0x7f60c8103ec0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.347+0000 7f60beffd700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60b4039200 0x7f60b403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.347+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f60c004ba60 con 0x7f60c8103ec0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.347+0000 7f60cd871700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60b4039200 0x7f60b403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:15.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.347+0000 7f60cd871700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60b4039200 0x7f60b403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60c8103380 tx=0x7f60c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:15.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.352+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f60c0010070 con 0x7f60c8103ec0 2024-03-16T10:58:15.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:15.557+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f60c80627d0 con 0x7f60b4039200 2024-03-16T10:58:15.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:15 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:15.465+0000 7fa1a4927200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:58:15.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:15 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:15.560+0000 7fa1a4927200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:58:16.043 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:15 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:15.791+0000 7f62d541a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:58:16.043 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:15 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:15.935+0000 7f62d541a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:58:16.254 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:16.233+0000 7fa1a4927200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:58:16.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:16.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:58:16.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:58:16.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:16.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:16.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:58:16.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:58:16.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:16.276+0000 7f62d541a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:16.366+0000 7f62d541a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:58:16.367 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:16.570 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:16.327+0000 7fa1a4927200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:58:16.570 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:16.470+0000 7fa1a4927200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:58:16.571 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:16.570+0000 7fa1a4927200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:58:16.754 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:16.568+0000 7f62d541a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:58:16.754 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:16 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:16.754+0000 7f62d541a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:58:16.916 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:16.654+0000 7fa1a4927200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:58:16.916 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:16.916+0000 7fa1a4927200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:17.025+0000 7fa1a4927200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:17.111+0000 7fa1a4927200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:17.204+0000 7fa1a4927200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:17.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:17.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:17.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:17.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:17.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:17.031+0000 7f62d541a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:17.289+0000 7f62d541a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:58:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:58:17.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:17.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:17.464 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:17 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:17.377+0000 7fa1a4927200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:58:17.543 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:17 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:17.520+0000 7f62d541a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:58:18.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:18 smithi078 ceph-mon[32235]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T10:58:18.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:18 smithi078 ceph-mon[32235]: Reconfiguring daemon mgr.x on smithi078 2024-03-16T10:58:18.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:18 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1046467149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543437a6-61d5-4969-be4e-7a20f1edc236"}]: dispatch 2024-03-16T10:58:18.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:18 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1046467149' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "543437a6-61d5-4969-be4e-7a20f1edc236"}]': finished 2024-03-16T10:58:18.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:18 smithi078 ceph-mon[32235]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T10:58:18.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:18.463 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:18.335+0000 7fa1a4927200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:58:18.463 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:18.463+0000 7fa1a4927200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:58:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-mon[35848]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T10:58:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-mon[35848]: Reconfiguring daemon mgr.x on smithi078 2024-03-16T10:58:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1046467149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543437a6-61d5-4969-be4e-7a20f1edc236"}]: dispatch 2024-03-16T10:58:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1046467149' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "543437a6-61d5-4969-be4e-7a20f1edc236"}]': finished 2024-03-16T10:58:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-mon[35848]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T10:58:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:18.543 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:18.306+0000 7f62d541a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:58:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-mon[35521]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T10:58:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-mon[35521]: Reconfiguring daemon mgr.x on smithi078 2024-03-16T10:58:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1046467149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543437a6-61d5-4969-be4e-7a20f1edc236"}]: dispatch 2024-03-16T10:58:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1046467149' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "543437a6-61d5-4969-be4e-7a20f1edc236"}]': finished 2024-03-16T10:58:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-mon[35521]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T10:58:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:19.072 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:18.791+0000 7fa1a4927200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:58:19.072 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:18 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:18.879+0000 7fa1a4927200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:58:19.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:19 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:19.071+0000 7fa1a4927200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:58:19.130 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:18.871+0000 7f62d541a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:58:19.130 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:18.966+0000 7f62d541a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:58:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:19 smithi098 ceph-mon[35848]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:19 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3743966464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:58:19.465 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:19 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:19.254+0000 7fa1a4927200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:58:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:19 smithi153 ceph-mon[35521]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:19 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3743966464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:58:19.544 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 10:58:19 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z[37374]: 2024-03-16T10:58:19.177+0000 7f62d541a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:58:19.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:19 smithi078 ceph-mon[32235]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:19.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:19 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3743966464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:58:19.791 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:19 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:19.534+0000 7fa1a4927200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:58:19.791 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:19 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:19.790+0000 7fa1a4927200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:58:20.131 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:20.022+0000 7fa1a4927200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:58:20.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:20.139+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f60c00155a0 con 0x7f60c8103ec0 2024-03-16T10:58:20.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-mon[35848]: Standby manager daemon z started 2024-03-16T10:58:20.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T10:58:20.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:58:20.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T10:58:20.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:58:20.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-mon[35848]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:20 smithi153 ceph-mon[35521]: Standby manager daemon z started 2024-03-16T10:58:20.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:20 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T10:58:20.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:20 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:58:20.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:20 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T10:58:20.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:20 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:58:20.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:20 smithi153 ceph-mon[35521]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:20 smithi078 ceph-mon[32235]: Standby manager daemon z started 2024-03-16T10:58:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:20 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T10:58:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:20 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:58:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:20 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T10:58:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:20 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.153:0/3241008531' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:58:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:20 smithi078 ceph-mon[32235]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:21.139 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:20 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:20.818+0000 7fa1a4927200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:58:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:21 smithi098 ceph-mon[35848]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-16T10:58:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T10:58:21.465 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:21 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:21.384+0000 7fa1a4927200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:58:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:21 smithi153 ceph-mon[35521]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-16T10:58:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T10:58:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:21 smithi078 ceph-mon[32235]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-16T10:58:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T10:58:21.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:21 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:21.480+0000 7fa1a4927200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:58:21.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 10:58:21 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y[37251]: 2024-03-16T10:58:21.692+0000 7fa1a4927200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:58:22.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:22.148+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f60c000db20 con 0x7f60c8103ec0 2024-03-16T10:58:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:22 smithi078 ceph-mon[32235]: Standby manager daemon y started 2024-03-16T10:58:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:22 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T10:58:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:22 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:58:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:22 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T10:58:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:22 smithi078 ceph-mon[32235]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:58:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:22 smithi078 ceph-mon[32235]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:22 smithi098 ceph-mon[35848]: Standby manager daemon y started 2024-03-16T10:58:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:22 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T10:58:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:22 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:58:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:22 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T10:58:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:22 smithi098 ceph-mon[35848]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:58:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:22 smithi098 ceph-mon[35848]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:22 smithi153 ceph-mon[35521]: Standby manager daemon y started 2024-03-16T10:58:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:22 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T10:58:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:22 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:58:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:22 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T10:58:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:22 smithi153 ceph-mon[35521]: from='mgr.? 172.21.15.98:0/2263991009' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:58:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:22 smithi153 ceph-mon[35521]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:23.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:23 smithi078 ceph-mon[32235]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-03-16T10:58:23.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:23 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T10:58:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:23 smithi098 ceph-mon[35848]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-03-16T10:58:23.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:23 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T10:58:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:23 smithi153 ceph-mon[35521]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-03-16T10:58:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:23 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T10:58:24.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:24.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:24.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:24.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:24.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:24.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:24.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:24 smithi078 ceph-mon[32235]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:24.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:24 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:24.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:24 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:24.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:24 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:24.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:24 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:24.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:24 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:24.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:24 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:24.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:24 smithi098 ceph-mon[35848]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:24 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:24 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:24 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:24 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:24 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:24 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:24 smithi153 ceph-mon[35521]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:27.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:26 smithi078 ceph-mon[32235]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:27.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T10:58:27.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:27.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:26 smithi098 ceph-mon[35848]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:27.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T10:58:27.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:27.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:26 smithi153 ceph-mon[35521]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:27.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T10:58:27.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:28.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:27 smithi078 ceph-mon[32235]: Deploying daemon osd.0 on smithi078 2024-03-16T10:58:28.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:27 smithi098 ceph-mon[35848]: Deploying daemon osd.0 on smithi078 2024-03-16T10:58:28.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:27 smithi153 ceph-mon[35521]: Deploying daemon osd.0 on smithi078 2024-03-16T10:58:29.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:28 smithi098 ceph-mon[35848]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:29.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:28 smithi078 ceph-mon[32235]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:29.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:28 smithi153 ceph-mon[35521]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:31.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:30 smithi098 ceph-mon[35848]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:31.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:30 smithi153 ceph-mon[35521]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:31.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:30 smithi078 ceph-mon[32235]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:32.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:32.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:32.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:32.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:32 smithi098 ceph-mon[35848]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:32.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:32.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:32.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:32.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:32 smithi078 ceph-mon[32235]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:32.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:32.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:32.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:32.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:32 smithi153 ceph-mon[35521]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:34.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:34 smithi078 ceph-mon[32235]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.019+0000 7f60beffd700 1 -- 172.21.15.78:0/3970603162 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f60c80627d0 con 0x7f60b4039200 2024-03-16T10:58:35.020 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2024-03-16T10:58:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60b4039200 msgr2=0x7f60b403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60b4039200 0x7f60b403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60c8103380 tx=0x7f60c400b410 comp rx=0 tx=0).stop 2024-03-16T10:58:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103ec0 msgr2=0x7f60c819fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103ec0 0x7f60c819fa20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f60c0007f40 tx=0x7f60c000d3b0 comp rx=0 tx=0).stop 2024-03-16T10:58:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 shutdown_connections 2024-03-16T10:58:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60b4039200 0x7f60b403b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c8102320 0x7f60c8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60c8103ec0 0x7f60c819fa20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 --2- 172.21.15.78:0/3970603162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60c8103520 0x7f60c81989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 >> 172.21.15.78:0/3970603162 conn(0x7f60c80fd8e0 msgr2=0x7f60c8106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 shutdown_connections 2024-03-16T10:58:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:58:35.021+0000 7f60cfad5700 1 -- 172.21.15.78:0/3970603162 wait complete. 2024-03-16T10:58:35.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:34 smithi098 ceph-mon[35848]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:35.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:34 smithi153 ceph-mon[35521]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:35.662 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.0.service 2024-03-16T10:58:35.664 INFO:tasks.cephadm:Deploying osd.1 on smithi098 with /dev/vg_nvme/lv_4... 2024-03-16T10:58:35.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T10:58:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:58:36.291 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.291 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.291 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.291 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.291 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:36 smithi098 ceph-mon[35848]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T10:58:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:36 smithi153 ceph-mon[35521]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T10:58:36.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:36.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:36 smithi078 ceph-mon[32235]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T10:58:37.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:37 smithi098 ceph-mon[35848]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:37.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:37 smithi098 ceph-mon[35848]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T10:58:37.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:37 smithi098 ceph-mon[35848]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T10:58:37.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:37 smithi098 ceph-mon[35848]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T10:58:37.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:37 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:37 smithi078 ceph-mon[32235]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:37 smithi078 ceph-mon[32235]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T10:58:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:37 smithi078 ceph-mon[32235]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T10:58:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:37 smithi078 ceph-mon[32235]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T10:58:37.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:37 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:37 smithi153 ceph-mon[35521]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:37 smithi153 ceph-mon[35521]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T10:58:37.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:37 smithi153 ceph-mon[35521]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T10:58:37.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:37 smithi153 ceph-mon[35521]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T10:58:37.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:37 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:37.815 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T10:58:37.815 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T10:58:37.816 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T10:58:37.816 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T10:58:37.816 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114469 s, 916 MB/s 2024-03-16T10:58:37.816 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T10:58:38.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-mon[32235]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T10:58:38.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-mon[32235]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T10:58:38.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-mon[32235]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:38.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-mon[32235]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' 2024-03-16T10:58:38.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.230 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0[46967]: 2024-03-16T10:58:38.019+0000 7f2fce51b700 -1 osd.0 0 waiting for initial osdmap 2024-03-16T10:58:38.230 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:58:38 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0[46967]: 2024-03-16T10:58:38.228+0000 7f2fc6306700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:58:38.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:38 smithi098 ceph-mon[35848]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T10:58:38.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:38 smithi098 ceph-mon[35848]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T10:58:38.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:38 smithi098 ceph-mon[35848]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:38.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:38 smithi098 ceph-mon[35848]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' 2024-03-16T10:58:38.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-03-16T10:58:38.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:38 smithi153 ceph-mon[35521]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T10:58:38.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:38 smithi153 ceph-mon[35521]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T10:58:38.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:38 smithi153 ceph-mon[35521]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:38.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:38 smithi153 ceph-mon[35521]: from='osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989]' entity='osd.0' 2024-03-16T10:58:38.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:58:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi078 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi078 to 17088M 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:39.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:39.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T10:58:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T10:58:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi078 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi078 to 17088M 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:39.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T10:58:39.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T10:58:39.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi078 2024-03-16T10:58:39.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi078 to 17088M 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:39.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.014+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/326385465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4102300 msgr2=0x7f7bd4102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.014+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/326385465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4102300 0x7f7bd4102720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7bc40099c0 tx=0x7f7bc4009cd0 comp rx=0 tx=0).stop 2024-03-16T10:58:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.014+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/326385465 shutdown_connections 2024-03-16T10:58:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.014+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/326385465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd4103ea0 0x7f7bd410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.014+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/326385465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4102300 0x7f7bd4102720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.014+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/326385465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd4103500 0x7f7bd4103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.014+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/326385465 >> 172.21.15.98:0/326385465 conn(0x7f7bd40fd8e0 msgr2=0x7f7bd40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.015+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/326385465 shutdown_connections 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.015+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/326385465 wait complete. 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.015+0000 7f7bd9b9f700 1 Processor -- start 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.015+0000 7f7bd9b9f700 1 -- start start 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.015+0000 7f7bd9b9f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd4102300 0x7f7bd41984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.015+0000 7f7bd9b9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd4103500 0x7f7bd4198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd9b9f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4103ea0 0x7f7bd419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd9b9f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd410ddf0 con 0x7f7bd4103500 2024-03-16T10:58:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd9b9f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd410dc70 con 0x7f7bd4103ea0 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd9b9f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd410df70 con 0x7f7bd4102300 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4103ea0 0x7f7bd419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd4103500 0x7f7bd4198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4103ea0 0x7f7bd419fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39564/0 (socket says 172.21.15.98:39564) 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd4103500 0x7f7bd4198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.98:39622/0 (socket says 172.21.15.98:39622) 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 -- 172.21.15.98:0/4266553968 learned_addr learned my addr 172.21.15.98:0/4266553968 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd37fe700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd4102300 0x7f7bd41984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 -- 172.21.15.98:0/4266553968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd4102300 msgr2=0x7f7bd41984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd4102300 0x7f7bd41984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 -- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd4103500 msgr2=0x7f7bd4198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd4103500 0x7f7bd4198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 -- 172.21.15.98:0/4266553968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd41a0120 con 0x7f7bd4103ea0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd3fff700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4103ea0 0x7f7bd419fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00ef30 tx=0x7f7bcc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd0ff9700 1 -- 172.21.15.98:0/4266553968 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc0092e0 con 0x7f7bd4103ea0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd0ff9700 1 -- 172.21.15.98:0/4266553968 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bcc004bb0 con 0x7f7bd4103ea0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd4074500 con 0x7f7bd4103ea0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd0ff9700 1 -- 172.21.15.98:0/4266553968 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc004020 con 0x7f7bd4103ea0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.016+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd4074a90 con 0x7f7bd4103ea0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.017+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd4068f90 con 0x7f7bd4103ea0 2024-03-16T10:58:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.018+0000 7f7bd0ff9700 1 -- 172.21.15.98:0/4266553968 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f7bcc01b030 con 0x7f7bd4103ea0 2024-03-16T10:58:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.019+0000 7f7bd0ff9700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7bbc0a2880 0x7f7bbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:58:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.019+0000 7f7bd0ff9700 1 -- 172.21.15.98:0/4266553968 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f7bcc0cc9c0 con 0x7f7bd4103ea0 2024-03-16T10:58:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.019+0000 7f7bd37fe700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7bbc0a2880 0x7f7bbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:58:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.020+0000 7f7bd37fe700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7bbc0a2880 0x7f7bbc0a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400b580 tx=0x7f7bc4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:58:40.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.024+0000 7f7bd0ff9700 1 -- 172.21.15.98:0/4266553968 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7bcc022020 con 0x7f7bd4103ea0 2024-03-16T10:58:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:40.226+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7bd40627d0 con 0x7f7bbc0a2880 2024-03-16T10:58:40.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:40 smithi098 ceph-mon[35848]: osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] boot 2024-03-16T10:58:40.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:40 smithi098 ceph-mon[35848]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T10:58:40.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:40 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:40.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:40 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:40.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:40 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:40.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:40 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:40.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:40 smithi098 ceph-mon[35848]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:40 smithi153 ceph-mon[35521]: osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] boot 2024-03-16T10:58:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:40 smithi153 ceph-mon[35521]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T10:58:40.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:40 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:40.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:40 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:40.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:40 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:40.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:40 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:40.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:40 smithi153 ceph-mon[35521]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:40 smithi078 ceph-mon[32235]: osd.0 [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] boot 2024-03-16T10:58:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:40 smithi078 ceph-mon[32235]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T10:58:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:58:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:58:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:40 smithi078 ceph-mon[32235]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:58:41.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:41 smithi153 ceph-mon[35521]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T10:58:41.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:41 smithi153 ceph-mon[35521]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:41.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:58:41.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:58:41.446 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:41 smithi098 ceph-mon[35848]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T10:58:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:41 smithi098 ceph-mon[35848]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:58:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:58:41.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:41 smithi078 ceph-mon[32235]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T10:58:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:41 smithi078 ceph-mon[32235]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:58:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:58:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:58:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:42.189 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:42 smithi098 ceph-mon[35848]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:42.461 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:42 smithi153 ceph-mon[35521]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:42.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:42 smithi078 ceph-mon[32235]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:43.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:43 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/4220172522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]: dispatch 2024-03-16T10:58:43.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:43 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]: dispatch 2024-03-16T10:58:43.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:43 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]': finished 2024-03-16T10:58:43.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:43 smithi098 ceph-mon[35848]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T10:58:43.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:43 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:58:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:43 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/4220172522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]: dispatch 2024-03-16T10:58:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:43 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]: dispatch 2024-03-16T10:58:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:43 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]': finished 2024-03-16T10:58:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:43 smithi153 ceph-mon[35521]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T10:58:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:43 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:58:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:43 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/4220172522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]: dispatch 2024-03-16T10:58:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:43 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]: dispatch 2024-03-16T10:58:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:43 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c57f283-922d-4ef4-a00d-ce2f62a7e19a"}]': finished 2024-03-16T10:58:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:43 smithi078 ceph-mon[32235]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T10:58:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:58:44.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/4190300772' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:58:44.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:44 smithi098 ceph-mon[35848]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:44.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/4190300772' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:58:44.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:44 smithi153 ceph-mon[35521]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/4190300772' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:58:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:44 smithi078 ceph-mon[32235]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:47.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:46 smithi098 ceph-mon[35848]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:46 smithi153 ceph-mon[35521]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:47.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:46 smithi078 ceph-mon[32235]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:49.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:48 smithi098 ceph-mon[35848]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:49.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:48 smithi078 ceph-mon[32235]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:49.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:48 smithi153 ceph-mon[35521]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:50.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T10:58:50.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T10:58:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T10:58:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:58:51.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:50 smithi098 ceph-mon[35848]: Deploying daemon osd.1 on smithi098 2024-03-16T10:58:51.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:50 smithi098 ceph-mon[35848]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:50 smithi153 ceph-mon[35521]: Deploying daemon osd.1 on smithi098 2024-03-16T10:58:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:50 smithi153 ceph-mon[35521]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:51.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:50 smithi078 ceph-mon[32235]: Deploying daemon osd.1 on smithi098 2024-03-16T10:58:51.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:50 smithi078 ceph-mon[32235]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:53.134 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:52 smithi098 ceph-mon[35848]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:52 smithi153 ceph-mon[35521]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:52 smithi078 ceph-mon[32235]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:55.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:54 smithi098 ceph-mon[35848]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:55.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:54 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:55.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:54 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:55.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:54 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:55.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:54 smithi153 ceph-mon[35521]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:55.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:54 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:55.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:54 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:55.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:54 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:55.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:54 smithi078 ceph-mon[32235]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:55.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:54 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:55.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:54 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:55.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:54 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:58:57.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:56 smithi098 ceph-mon[35848]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:57.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:56 smithi153 ceph-mon[35521]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:56 smithi078 ceph-mon[32235]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:57.989 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 1 on host 'smithi098' 2024-03-16T10:58:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.988+0000 7f7bd0ff9700 1 -- 172.21.15.98:0/4266553968 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7bd40627d0 con 0x7f7bbc0a2880 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7bbc0a2880 msgr2=0x7f7bbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7bbc0a2880 0x7f7bbc0a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400b580 tx=0x7f7bc4005fb0 comp rx=0 tx=0).stop 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4103ea0 msgr2=0x7f7bd419fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4103ea0 0x7f7bd419fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00ef30 tx=0x7f7bcc00c3a0 comp rx=0 tx=0).stop 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 shutdown_connections 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7bbc0a2880 0x7f7bbc0a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd4102300 0x7f7bd41984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd4103ea0 0x7f7bd419fa80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 --2- 172.21.15.98:0/4266553968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd4103500 0x7f7bd4198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 >> 172.21.15.98:0/4266553968 conn(0x7f7bd40fd8e0 msgr2=0x7f7bd4106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 shutdown_connections 2024-03-16T10:58:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:58:57.990+0000 7f7bd9b9f700 1 -- 172.21.15.98:0/4266553968 wait complete. 2024-03-16T10:58:58.968 DEBUG:teuthology.orchestra.run.smithi098:osd.1> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.1.service 2024-03-16T10:58:58.970 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_4... 2024-03-16T10:58:58.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T10:58:59.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:58 smithi098 ceph-mon[35848]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:59.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:58 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:58 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:58 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:58 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:58 smithi098 ceph-mon[35848]: from='osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:58:59.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:58:58 smithi098 ceph-mon[35848]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:58:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:58:59.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:58 smithi153 ceph-mon[35521]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:58 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:58 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:58 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:58 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:58 smithi153 ceph-mon[35521]: from='osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:58:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:58:58 smithi153 ceph-mon[35521]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:58:59.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:58 smithi078 ceph-mon[32235]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:58:59.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:58 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:58 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:58 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:58 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:58:59.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:58 smithi078 ceph-mon[32235]: from='osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:58:59.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:58:58 smithi078 ceph-mon[32235]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:59:00.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:00 smithi153 ceph-mon[35521]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T10:59:00.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:00 smithi153 ceph-mon[35521]: from='osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T10:59:00.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:00 smithi153 ceph-mon[35521]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T10:59:00.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:00.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:00 smithi153 ceph-mon[35521]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T10:59:00.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:00 smithi153 ceph-mon[35521]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:59:00.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:00 smithi098 ceph-mon[35848]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T10:59:00.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:00 smithi098 ceph-mon[35848]: from='osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T10:59:00.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:00 smithi098 ceph-mon[35848]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T10:59:00.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:00.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:00 smithi098 ceph-mon[35848]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T10:59:00.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:00 smithi098 ceph-mon[35848]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:59:01.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:00 smithi078 ceph-mon[32235]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T10:59:01.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:00 smithi078 ceph-mon[32235]: from='osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T10:59:01.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:00 smithi078 ceph-mon[32235]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T10:59:01.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:01.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:00 smithi078 ceph-mon[32235]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T10:59:01.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:00 smithi078 ceph-mon[32235]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:59:01.079 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T10:59:01.079 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T10:59:01.079 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T10:59:01.079 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T10:59:01.079 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121507 s, 863 MB/s 2024-03-16T10:59:01.079 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T10:59:01.657 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:01 smithi153 ceph-mon[35521]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T10:59:01.658 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:01 smithi153 ceph-mon[35521]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T10:59:01.658 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:01.658 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:01.658 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:01 smithi153 ceph-mon[35521]: from='osd.1 ' entity='osd.1' 2024-03-16T10:59:01.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-03-16T10:59:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:01 smithi098 ceph-mon[35848]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T10:59:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:01 smithi098 ceph-mon[35848]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T10:59:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:01 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:01 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:01 smithi098 ceph-mon[35848]: from='osd.1 ' entity='osd.1' 2024-03-16T10:59:01.715 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1[40414]: 2024-03-16T10:59:01.528+0000 7fe18cc8a700 -1 osd.1 0 waiting for initial osdmap 2024-03-16T10:59:01.715 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1[40414]: 2024-03-16T10:59:01.633+0000 7fe184a75700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:59:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:59:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:01 smithi078 ceph-mon[32235]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T10:59:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:01 smithi078 ceph-mon[32235]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T10:59:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:01 smithi078 ceph-mon[32235]: from='osd.1 ' entity='osd.1' 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi098 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:02.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:02.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:02.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:02.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:02.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T10:59:02.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T10:59:02.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi098 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:02.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:03.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T10:59:03.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T10:59:03.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:03.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:59:03.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi098 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:03.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:03.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 -- 172.21.15.153:0/3107473345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254101210 msgr2=0x7ff254101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:03.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/3107473345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254101210 0x7ff254101630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff244009990 tx=0x7ff244009ca0 comp rx=0 tx=0).stop 2024-03-16T10:59:03.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 -- 172.21.15.153:0/3107473345 shutdown_connections 2024-03-16T10:59:03.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/3107473345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254101210 0x7ff254101630 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/3107473345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff254102db0 0x7ff254109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/3107473345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff254102410 0x7ff254102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 -- 172.21.15.153:0/3107473345 >> 172.21.15.153:0/3107473345 conn(0x7ff2540fc7d0 msgr2=0x7ff2540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:59:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 -- 172.21.15.153:0/3107473345 shutdown_connections 2024-03-16T10:59:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 -- 172.21.15.153:0/3107473345 wait complete. 2024-03-16T10:59:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 Processor -- start 2024-03-16T10:59:03.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.444+0000 7ff25bfcd700 1 -- start start 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25bfcd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff254101210 0x7ff254197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25bfcd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff254102410 0x7ff2541978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25bfcd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254102db0 0x7ff25419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25bfcd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff25410ccb0 con 0x7ff254102410 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25bfcd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff25410cb30 con 0x7ff254101210 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25bfcd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff25410ce30 con 0x7ff254102db0 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254102db0 0x7ff25419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254102db0 0x7ff25419e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52372/0 (socket says 172.21.15.153:52372) 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 -- 172.21.15.153:0/1609633713 learned_addr learned my addr 172.21.15.153:0/1609633713 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff259568700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff254102410 0x7ff2541978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:03.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff259d69700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff254101210 0x7ff254197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 -- 172.21.15.153:0/1609633713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff254101210 msgr2=0x7ff254197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff254101210 0x7ff254197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 -- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff254102410 msgr2=0x7ff2541978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff254102410 0x7ff2541978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 -- 172.21.15.153:0/1609633713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25419efe0 con 0x7ff254102db0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25a56a700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254102db0 0x7ff25419e940 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff24c007bd0 tx=0x7ff24c007ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff24affd700 1 -- 172.21.15.153:0/1609633713 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c016030 con 0x7ff254102db0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff24affd700 1 -- 172.21.15.153:0/1609633713 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff24c00fe90 con 0x7ff254102db0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.445+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25406aa90 con 0x7ff254102db0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.446+0000 7ff24affd700 1 -- 172.21.15.153:0/1609633713 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c005020 con 0x7ff254102db0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.446+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25406b020 con 0x7ff254102db0 2024-03-16T10:59:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.446+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff254068f90 con 0x7ff254102db0 2024-03-16T10:59:03.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.448+0000 7ff24affd700 1 -- 172.21.15.153:0/1609633713 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7ff24c00f9e0 con 0x7ff254102db0 2024-03-16T10:59:03.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.449+0000 7ff24affd700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2400a2880 0x7ff2400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:03.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.449+0000 7ff24affd700 1 -- 172.21.15.153:0/1609633713 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7ff24c0cf710 con 0x7ff254102db0 2024-03-16T10:59:03.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.449+0000 7ff259d69700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2400a2880 0x7ff2400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:03.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.449+0000 7ff259d69700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2400a2880 0x7ff2400a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff2440037f0 tx=0x7ff244005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:59:03.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.453+0000 7ff24affd700 1 -- 172.21.15.153:0/1609633713 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff24c09e000 con 0x7ff254102db0 2024-03-16T10:59:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:03.656+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff25406b280 con 0x7ff2400a2880 2024-03-16T10:59:03.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:03 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi098 to 17088M 2024-03-16T10:59:03.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:03 smithi098 ceph-mon[35848]: osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] boot 2024-03-16T10:59:03.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:03 smithi098 ceph-mon[35848]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T10:59:03.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:03 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi098 to 17088M 2024-03-16T10:59:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:03 smithi153 ceph-mon[35521]: osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] boot 2024-03-16T10:59:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:03 smithi153 ceph-mon[35521]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T10:59:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:03 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi098 to 17088M 2024-03-16T10:59:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:03 smithi078 ceph-mon[32235]: osd.1 [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] boot 2024-03-16T10:59:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:03 smithi078 ceph-mon[32235]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T10:59:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:59:04.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:04 smithi098 ceph-mon[35848]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T10:59:04.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:04 smithi098 ceph-mon[35848]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:04.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:04.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:04.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:04.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:04 smithi098 ceph-mon[35848]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:05.031 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:04 smithi153 ceph-mon[35521]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T10:59:05.031 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:04 smithi153 ceph-mon[35521]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:05.032 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:05.032 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:05.032 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:05.032 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:04 smithi153 ceph-mon[35521]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:05.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:04 smithi078 ceph-mon[32235]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T10:59:05.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:04 smithi078 ceph-mon[32235]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:05.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:05.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:05.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:05.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:04 smithi078 ceph-mon[32235]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:07 smithi153 ceph-mon[35521]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:07 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/961038347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]: dispatch 2024-03-16T10:59:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:07 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]: dispatch 2024-03-16T10:59:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:07 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]': finished 2024-03-16T10:59:07.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:07 smithi153 ceph-mon[35521]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T10:59:07.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:07 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:07.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:07 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/1248869487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:06 smithi078 ceph-mon[32235]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:06 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/961038347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]: dispatch 2024-03-16T10:59:07.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:06 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]: dispatch 2024-03-16T10:59:07.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:06 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]': finished 2024-03-16T10:59:07.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:06 smithi078 ceph-mon[32235]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T10:59:07.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:07.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:06 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/1248869487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:07.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:06 smithi098 ceph-mon[35848]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:07.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:06 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/961038347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]: dispatch 2024-03-16T10:59:07.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:06 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]: dispatch 2024-03-16T10:59:07.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:06 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4559999e-89a9-433d-b3b8-bdec51aebeb8"}]': finished 2024-03-16T10:59:07.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:06 smithi098 ceph-mon[35848]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T10:59:07.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:07.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:06 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/1248869487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:08 smithi153 ceph-mon[35521]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:08 smithi078 ceph-mon[32235]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:08 smithi098 ceph-mon[35848]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:11.251 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:10 smithi153 ceph-mon[35521]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:11.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:10 smithi078 ceph-mon[32235]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:10 smithi098 ceph-mon[35848]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:13.254 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:12 smithi153 ceph-mon[35521]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:13.267 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:12 smithi098 ceph-mon[35848]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:12 smithi078 ceph-mon[32235]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:14.224 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T10:59:14.224 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:14.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T10:59:14.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:14.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T10:59:14.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:15.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:14 smithi153 ceph-mon[35521]: Deploying daemon osd.2 on smithi153 2024-03-16T10:59:15.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:14 smithi153 ceph-mon[35521]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:15.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:14 smithi078 ceph-mon[32235]: Deploying daemon osd.2 on smithi153 2024-03-16T10:59:15.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:14 smithi078 ceph-mon[32235]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:14 smithi098 ceph-mon[35848]: Deploying daemon osd.2 on smithi153 2024-03-16T10:59:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:14 smithi098 ceph-mon[35848]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:17.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:16 smithi153 ceph-mon[35521]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:17.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:16 smithi078 ceph-mon[32235]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:16 smithi098 ceph-mon[35848]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:19.252 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:18 smithi153 ceph-mon[35521]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:19.252 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:19.252 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:19.252 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:19.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:18 smithi078 ceph-mon[32235]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:19.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:19.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:19.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:19.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:19 smithi098 ceph-mon[35848]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:19.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:19 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:19.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:19 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:19.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:19 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:21.242 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:21 smithi153 ceph-mon[35521]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:21.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:21 smithi078 ceph-mon[32235]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:21 smithi098 ceph-mon[35848]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:22.326 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-03-16T10:59:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.325+0000 7ff24affd700 1 -- 172.21.15.153:0/1609633713 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff25406b280 con 0x7ff2400a2880 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.327+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2400a2880 msgr2=0x7ff2400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2400a2880 0x7ff2400a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff2440037f0 tx=0x7ff244005fd0 comp rx=0 tx=0).stop 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254102db0 msgr2=0x7ff25419e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254102db0 0x7ff25419e940 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff24c007bd0 tx=0x7ff24c007ee0 comp rx=0 tx=0).stop 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 shutdown_connections 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2400a2880 0x7ff2400a4d40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff254102db0 0x7ff25419e940 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff254101210 0x7ff254197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 --2- 172.21.15.153:0/1609633713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff254102410 0x7ff2541978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 >> 172.21.15.153:0/1609633713 conn(0x7ff2540fc7d0 msgr2=0x7ff254105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 shutdown_connections 2024-03-16T10:59:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T10:59:22.328+0000 7ff25bfcd700 1 -- 172.21.15.153:0/1609633713 wait complete. 2024-03-16T10:59:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:59:22.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:22 smithi153 ceph-mon[35521]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:59:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:59:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:22 smithi078 ceph-mon[32235]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:59:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:22 smithi098 ceph-mon[35848]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:59:23.025 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.2.service 2024-03-16T10:59:23.027 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_3... 2024-03-16T10:59:23.027 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T10:59:23.272 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:59:24.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-mon[35521]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T10:59:24.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-mon[35521]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T10:59:24.441 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-mon[35521]: from='osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T10:59:24.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:24.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-mon[35521]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T10:59:24.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-mon[35521]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:24.442 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2[40666]: 2024-03-16T10:59:24.298+0000 7f13f674e700 -1 osd.2 0 waiting for initial osdmap 2024-03-16T10:59:24.442 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2[40666]: 2024-03-16T10:59:24.302+0000 7f13f2541700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:59:24.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:24 smithi078 ceph-mon[32235]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T10:59:24.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:24 smithi078 ceph-mon[32235]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T10:59:24.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:24 smithi078 ceph-mon[32235]: from='osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T10:59:24.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:24 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:24.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:24 smithi078 ceph-mon[32235]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T10:59:24.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:24 smithi078 ceph-mon[32235]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:24.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:24 smithi098 ceph-mon[35848]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T10:59:24.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:24 smithi098 ceph-mon[35848]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T10:59:24.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:24 smithi098 ceph-mon[35848]: from='osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T10:59:24.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:24 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:24.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:24 smithi098 ceph-mon[35848]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T10:59:24.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:24 smithi098 ceph-mon[35848]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:25.263 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T10:59:25.263 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T10:59:25.263 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-16T10:59:25.263 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-16T10:59:25.263 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130157 s, 806 MB/s 2024-03-16T10:59:25.263 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-16T10:59:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:25 smithi078 ceph-mon[32235]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T10:59:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:25 smithi078 ceph-mon[32235]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T10:59:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:25 smithi153 ceph-mon[35521]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T10:59:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:25 smithi153 ceph-mon[35521]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T10:59:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:25 smithi098 ceph-mon[35848]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T10:59:25.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:25 smithi098 ceph-mon[35848]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T10:59:25.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:25.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2024-03-16T10:59:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] boot 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi153 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:26.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi153 to 17088M 2024-03-16T10:59:26.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:26.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:26.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:26.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:59:26.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] boot 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi153 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi153 to 17088M 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:26.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:59:26.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: osd.2 [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] boot 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi153 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi153 to 17088M 2024-03-16T10:59:26.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:26.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:26.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:26.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:59:26.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:59:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:27 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T10:59:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:27 smithi078 ceph-mon[32235]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T10:59:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:27 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:59:27.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:27 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T10:59:27.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:27 smithi098 ceph-mon[35848]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T10:59:27.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:27 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:59:27.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:27 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T10:59:27.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:27 smithi153 ceph-mon[35521]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T10:59:27.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:27 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:59:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 -- 172.21.15.78:0/1617668096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8108060 msgr2=0x7f06a810a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 --2- 172.21.15.78:0/1617668096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8108060 0x7f06a810a450 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f06a000b0a0 tx=0x7f06a000b3b0 comp rx=0 tx=0).stop 2024-03-16T10:59:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 -- 172.21.15.78:0/1617668096 shutdown_connections 2024-03-16T10:59:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 --2- 172.21.15.78:0/1617668096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8108060 0x7f06a810a450 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 --2- 172.21.15.78:0/1617668096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a8102e00 0x7f06a81051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 --2- 172.21.15.78:0/1617668096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8105730 0x7f06a8107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 -- 172.21.15.78:0/1617668096 >> 172.21.15.78:0/1617668096 conn(0x7f06a80fc770 msgr2=0x7f06a80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:59:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 -- 172.21.15.78:0/1617668096 shutdown_connections 2024-03-16T10:59:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.812+0000 7f06acc18700 1 -- 172.21.15.78:0/1617668096 wait complete. 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 Processor -- start 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 -- start start 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8102e00 0x7f06a8102760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8105730 0x7f06a8100db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a8108060 0x7f06a81012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f06a810dd50 con 0x7f06a8102e00 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06a810dbd0 con 0x7f06a8108060 2024-03-16T10:59:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06acc18700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06a810ded0 con 0x7f06a8105730 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06a659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8102e00 0x7f06a8102760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06a659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8102e00 0x7f06a8102760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44798/0 (socket says 172.21.15.78:44798) 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.813+0000 7f06a659c700 1 -- 172.21.15.78:0/749630726 learned_addr learned my addr 172.21.15.78:0/749630726 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a5d9b700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8105730 0x7f06a8100db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a6d9d700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a8108060 0x7f06a81012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a659c700 1 -- 172.21.15.78:0/749630726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8105730 msgr2=0x7f06a8100db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a659c700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8105730 0x7f06a8100db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a659c700 1 -- 172.21.15.78:0/749630726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a8108060 msgr2=0x7f06a81012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a659c700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a8108060 0x7f06a81012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a659c700 1 -- 172.21.15.78:0/749630726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a8101ba0 con 0x7f06a8102e00 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a6d9d700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a8108060 0x7f06a81012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06a659c700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8102e00 0x7f06a8102760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f069c00c920 tx=0x7f069c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:59:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c007b00 con 0x7f06a8102e00 2024-03-16T10:59:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f069c004ce0 con 0x7f06a8102e00 2024-03-16T10:59:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a81a25b0 con 0x7f06a8102e00 2024-03-16T10:59:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c005560 con 0x7f06a8102e00 2024-03-16T10:59:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.814+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a81a2ac0 con 0x7f06a8102e00 2024-03-16T10:59:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.815+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a8068f90 con 0x7f06a8102e00 2024-03-16T10:59:27.818 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51664]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:59:27.818 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:27.818 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51664]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:27.818 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51664]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.816+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f069c004750 con 0x7f06a8102e00 2024-03-16T10:59:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.817+0000 7f06977fe700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06900a2800 0x7f06900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.817+0000 7f06a5d9b700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06900a2800 0x7f06900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.817+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f069c0cd5d0 con 0x7f06a8102e00 2024-03-16T10:59:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.818+0000 7f06a5d9b700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06900a2800 0x7f06900a4cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0698005950 tx=0x7f06980058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:59:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:27.824+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f069c014030 con 0x7f06a8102e00 2024-03-16T10:59:28.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:28.034+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f06a8062710 con 0x7f06900a2800 2024-03-16T10:59:28.309 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51689]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:59:28.309 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51689]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:28.309 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51689]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:28.309 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 10:59:27 smithi078 sudo[51689]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:28.464 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44526]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:59:28.464 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44526]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:28.464 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44526]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:28.464 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44526]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:28.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:59:28.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T10:59:28.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:28.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:28.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:28.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:28.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:28.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:28.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:59:28.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T10:59:28.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:28.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:28.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:28.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:28.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:28.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:28.714 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:59:28.714 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:28.714 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44529]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:28.715 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 10:59:28 smithi098 sudo[44529]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:28.732 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:59:28.732 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T10:59:28.732 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:28.733 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:28.733 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:28.733 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:28.733 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:28.733 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:28.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44577]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:59:28.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44577]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:28.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44577]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:28.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44577]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:29.293 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44580]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:59:29.293 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44580]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:29.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44580]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:29.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 10:59:28 smithi153 sudo[44580]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:29.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51835]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:59:29.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51835]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:29.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51835]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:29.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51835]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:29.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:29.528+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f069c097a90 con 0x7f06a8102e00 2024-03-16T10:59:29.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 ceph-mon[35521]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T10:59:29.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:29.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:29.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 ceph-mon[35521]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:29.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 sudo[44583]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:59:29.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 sudo[44583]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:29.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 sudo[44583]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:29.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:29 smithi153 sudo[44583]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51838]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:59:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51838]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51838]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 sudo[51838]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 ceph-mon[32235]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T10:59:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:29.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:29 smithi078 ceph-mon[32235]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:29.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:29 smithi098 ceph-mon[35848]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T10:59:29.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:29.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:29.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:29 smithi098 ceph-mon[35848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:30.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 sudo[44586]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:59:30.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 sudo[44586]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:30.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 sudo[44586]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:30.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 sudo[44586]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:30.512 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44532]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:59:30.512 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44532]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:30.512 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44532]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:30.512 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44532]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:30.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:30.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:30 smithi153 ceph-mon[35521]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:30.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:30.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:30 smithi078 ceph-mon[32235]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44535]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:59:30.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:59:30.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44535]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:59:30.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 sudo[44535]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:59:30.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:59:30.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:59:30.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:59:30.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:30 smithi098 ceph-mon[35848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:59:31.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:31 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2632011338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a68ec8-b4f9-42fb-9f85-e683ef538fbe"}]: dispatch 2024-03-16T10:59:31.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:31 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2632011338' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3a68ec8-b4f9-42fb-9f85-e683ef538fbe"}]': finished 2024-03-16T10:59:31.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:31 smithi153 ceph-mon[35521]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T10:59:31.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:31.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:31 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1200225749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:31 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2632011338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a68ec8-b4f9-42fb-9f85-e683ef538fbe"}]: dispatch 2024-03-16T10:59:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:31 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2632011338' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3a68ec8-b4f9-42fb-9f85-e683ef538fbe"}]': finished 2024-03-16T10:59:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:31 smithi078 ceph-mon[32235]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T10:59:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:31 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1200225749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:31.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:31 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2632011338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a68ec8-b4f9-42fb-9f85-e683ef538fbe"}]: dispatch 2024-03-16T10:59:31.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:31 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2632011338' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3a68ec8-b4f9-42fb-9f85-e683ef538fbe"}]': finished 2024-03-16T10:59:31.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:31 smithi098 ceph-mon[35848]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T10:59:31.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:31.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:31 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1200225749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:32 smithi078 ceph-mon[32235]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:32.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:32 smithi098 ceph-mon[35848]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:33.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:32 smithi153 ceph-mon[35521]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:35.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:34 smithi098 ceph-mon[35848]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-16T10:59:35.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:34 smithi153 ceph-mon[35521]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-16T10:59:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:34 smithi078 ceph-mon[32235]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-16T10:59:37.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:36 smithi078 ceph-mon[32235]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-16T10:59:37.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:36 smithi153 ceph-mon[35521]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-16T10:59:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:36 smithi098 ceph-mon[35848]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-16T10:59:39.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:38 smithi078 ceph-mon[32235]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 191 KiB/s wr, 12 op/s 2024-03-16T10:59:39.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T10:59:39.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:39.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:38 smithi078 ceph-mon[32235]: Deploying daemon osd.3 on smithi078 2024-03-16T10:59:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:38 smithi153 ceph-mon[35521]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 191 KiB/s wr, 12 op/s 2024-03-16T10:59:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T10:59:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:39.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:38 smithi153 ceph-mon[35521]: Deploying daemon osd.3 on smithi078 2024-03-16T10:59:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:38 smithi098 ceph-mon[35848]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 191 KiB/s wr, 12 op/s 2024-03-16T10:59:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T10:59:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:38 smithi098 ceph-mon[35848]: Deploying daemon osd.3 on smithi078 2024-03-16T10:59:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:40 smithi153 ceph-mon[35521]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-16T10:59:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:40 smithi078 ceph-mon[32235]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-16T10:59:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:40 smithi098 ceph-mon[35848]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-16T10:59:43.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:42 smithi078 ceph-mon[32235]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-16T10:59:43.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:42 smithi098 ceph-mon[35848]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-16T10:59:43.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:42 smithi153 ceph-mon[35521]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-16T10:59:44.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:44 smithi098 ceph-mon[35848]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-16T10:59:45.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:45.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:45.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:45.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:44 smithi153 ceph-mon[35521]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-16T10:59:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:44 smithi078 ceph-mon[32235]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-16T10:59:47.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:46 smithi078 ceph-mon[32235]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:47.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:47.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:47.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.288+0000 7f06977fe700 1 -- 172.21.15.78:0/749630726 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f06a8062710 con 0x7f06900a2800 2024-03-16T10:59:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2024-03-16T10:59:47.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06900a2800 msgr2=0x7f06900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06900a2800 0x7f06900a4cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0698005950 tx=0x7f06980058e0 comp rx=0 tx=0).stop 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8102e00 msgr2=0x7f06a8102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8102e00 0x7f06a8102760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f069c00c920 tx=0x7f069c00cce0 comp rx=0 tx=0).stop 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 shutdown_connections 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06900a2800 0x7f06900a4cc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a8108060 0x7f06a81012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a8102e00 0x7f06a8102760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 --2- 172.21.15.78:0/749630726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8105730 0x7f06a8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 >> 172.21.15.78:0/749630726 conn(0x7f06a80fc770 msgr2=0x7f06a8103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 shutdown_connections 2024-03-16T10:59:47.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T10:59:47.291+0000 7f06acc18700 1 -- 172.21.15.78:0/749630726 wait complete. 2024-03-16T10:59:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:46 smithi153 ceph-mon[35521]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:46 smithi098 ceph-mon[35848]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:47.940 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.3.service 2024-03-16T10:59:47.942 INFO:tasks.cephadm:Deploying osd.4 on smithi098 with /dev/vg_nvme/lv_3... 2024-03-16T10:59:47.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T10:59:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:59:48.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:48 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:48.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:48 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:48 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:48 smithi078 ceph-mon[32235]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T10:59:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:48 smithi078 ceph-mon[32235]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:48 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:48 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:48 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:48 smithi153 ceph-mon[35521]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T10:59:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:48 smithi153 ceph-mon[35521]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:48.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:48 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:48.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:48 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:48.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:48 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:59:48.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:48 smithi098 ceph-mon[35848]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T10:59:48.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:48 smithi098 ceph-mon[35848]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:49.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:49 smithi078 ceph-mon[32235]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T10:59:49.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:49 smithi078 ceph-mon[32235]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T10:59:49.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:49 smithi078 ceph-mon[32235]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T10:59:49.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:49 smithi098 ceph-mon[35848]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T10:59:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:49 smithi098 ceph-mon[35848]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T10:59:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:49 smithi098 ceph-mon[35848]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T10:59:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:49 smithi153 ceph-mon[35521]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T10:59:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:49 smithi153 ceph-mon[35521]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T10:59:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:49 smithi153 ceph-mon[35521]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T10:59:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T10:59:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T10:59:50.036 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T10:59:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T10:59:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116006 s, 904 MB/s 2024-03-16T10:59:50.036 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 2024-03-16T10:59:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T10:59:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:50 smithi153 ceph-mon[35521]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T10:59:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:50 smithi153 ceph-mon[35521]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T10:59:50.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:50.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:50.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:50 smithi153 ceph-mon[35521]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:50.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:50 smithi153 ceph-mon[35521]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' 2024-03-16T10:59:50.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-mon[32235]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T10:59:50.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-mon[32235]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T10:59:50.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:50.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:50.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-mon[32235]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:50.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-mon[32235]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' 2024-03-16T10:59:50.560 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3[53953]: 2024-03-16T10:59:50.224+0000 7f5d77b0c700 -1 osd.3 0 waiting for initial osdmap 2024-03-16T10:59:50.560 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 10:59:50 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3[53953]: 2024-03-16T10:59:50.407+0000 7f5d6f8f7700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:59:50.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:50 smithi098 ceph-mon[35848]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T10:59:50.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:50 smithi098 ceph-mon[35848]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T10:59:50.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:50.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:50.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:50 smithi098 ceph-mon[35848]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:50.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:50 smithi098 ceph-mon[35848]: from='osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309]' entity='osd.3' 2024-03-16T10:59:50.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-03-16T10:59:50.880 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:59:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: osdmap e25: 4 total, 3 up, 4 in 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi078 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi078 to 8544M 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:51.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:51.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] boot 2024-03-16T10:59:51.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T10:59:51.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:51.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T10:59:51.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T10:59:51.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: osdmap e25: 4 total, 3 up, 4 in 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi078 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi078 to 8544M 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:51.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] boot 2024-03-16T10:59:51.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T10:59:51.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:51.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T10:59:51.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T10:59:51.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: osdmap e25: 4 total, 3 up, 4 in 2024-03-16T10:59:51.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:51.648 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi078 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi078 to 8544M 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: osd.3 [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] boot 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T10:59:51.649 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:59:52.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:52 smithi098 ceph-mon[35848]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 -- 172.21.15.98:0/2356652646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 msgr2=0x7f3ba0102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 --2- 172.21.15.98:0/2356652646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba0102740 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3b900099c0 tx=0x7f3b90009cd0 comp rx=0 tx=0).stop 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 -- 172.21.15.98:0/2356652646 shutdown_connections 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 --2- 172.21.15.98:0/2356652646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ba0103ec0 0x7f3ba010a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 --2- 172.21.15.98:0/2356652646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba0102740 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 --2- 172.21.15.98:0/2356652646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba0103520 0x7f3ba0103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 -- 172.21.15.98:0/2356652646 >> 172.21.15.98:0/2356652646 conn(0x7f3ba00fd8e0 msgr2=0x7f3ba00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 -- 172.21.15.98:0/2356652646 shutdown_connections 2024-03-16T10:59:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.444+0000 7f3ba5092700 1 -- 172.21.15.98:0/2356652646 wait complete. 2024-03-16T10:59:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 Processor -- start 2024-03-16T10:59:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 -- start start 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba0103520 0x7f3ba0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ba0103ec0 0x7f3ba019f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba010dda0 con 0x7f3ba0103520 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba010dc20 con 0x7f3ba0102320 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3ba5092700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba010df20 con 0x7f3ba0103ec0 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba01983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60040/0 (socket says 172.21.15.98:60040) 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 -- 172.21.15.98:0/3775255271 learned_addr learned my addr 172.21.15.98:0/3775255271 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9f59e700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ba0103ec0 0x7f3ba019f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9659c700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba0103520 0x7f3ba0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 -- 172.21.15.98:0/3775255271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ba0103ec0 msgr2=0x7f3ba019f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ba0103ec0 0x7f3ba019f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 -- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba0103520 msgr2=0x7f3ba0198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba0103520 0x7f3ba0198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.445+0000 7f3b9ed9d700 1 -- 172.21.15.98:0/3775255271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba01a0040 con 0x7f3ba0102320 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.446+0000 7f3b9ed9d700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba01983e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3b900098e0 tx=0x7f3b900057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.446+0000 7f3b9cd99700 1 -- 172.21.15.98:0/3775255271 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90027070 con 0x7f3ba0102320 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.446+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba01a02d0 con 0x7f3ba0102320 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.446+0000 7f3b9cd99700 1 -- 172.21.15.98:0/3775255271 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b90004050 con 0x7f3ba0102320 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.446+0000 7f3b9cd99700 1 -- 172.21.15.98:0/3775255271 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90022dd0 con 0x7f3ba0102320 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.446+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba01a0780 con 0x7f3ba0102320 2024-03-16T10:59:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.447+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba0068f90 con 0x7f3ba0102320 2024-03-16T10:59:52.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.448+0000 7f3b9cd99700 1 -- 172.21.15.98:0/3775255271 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3b9002a070 con 0x7f3ba0102320 2024-03-16T10:59:52.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.448+0000 7f3b9cd99700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b8c0a2890 0x7f3b8c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:59:52.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.449+0000 7f3b9cd99700 1 -- 172.21.15.98:0/3775255271 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f3b900ce6c0 con 0x7f3ba0102320 2024-03-16T10:59:52.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.449+0000 7f3b9659c700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b8c0a2890 0x7f3b8c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:59:52.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.449+0000 7f3b9659c700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b8c0a2890 0x7f3b8c0a4d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b88005950 tx=0x7f3b880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:59:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.453+0000 7f3b9cd99700 1 -- 172.21.15.98:0/3775255271 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b9005e740 con 0x7f3ba0102320 2024-03-16T10:59:52.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:52 smithi153 ceph-mon[35521]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:52.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:52 smithi078 ceph-mon[32235]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:59:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T10:59:52.655+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3ba0062710 con 0x7f3b8c0a2890 2024-03-16T10:59:53.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:53 smithi098 ceph-mon[35848]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T10:59:53.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:53 smithi098 ceph-mon[35848]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:53.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:53.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:53.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:53.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:53 smithi098 ceph-mon[35848]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T10:59:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:53 smithi153 ceph-mon[35521]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T10:59:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:53 smithi153 ceph-mon[35521]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:53 smithi153 ceph-mon[35521]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T10:59:53.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:53 smithi078 ceph-mon[32235]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T10:59:53.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:53 smithi078 ceph-mon[32235]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:59:53.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:59:53.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:59:53.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:59:53.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:53 smithi078 ceph-mon[32235]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T10:59:54.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:54 smithi098 ceph-mon[35848]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T10:59:54.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:54 smithi098 ceph-mon[35848]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-16T10:59:54.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:54 smithi153 ceph-mon[35521]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T10:59:54.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:54 smithi153 ceph-mon[35521]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-16T10:59:54.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:54 smithi078 ceph-mon[32235]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T10:59:54.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:54 smithi078 ceph-mon[32235]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-16T10:59:55.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:55 smithi098 ceph-mon[35848]: osdmap e30: 4 total, 4 up, 4 in 2024-03-16T10:59:55.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:55 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/4105508442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]: dispatch 2024-03-16T10:59:55.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:55 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]: dispatch 2024-03-16T10:59:55.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:55 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]': finished 2024-03-16T10:59:55.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:55 smithi098 ceph-mon[35848]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T10:59:55.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:55 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:59:55.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:55 smithi153 ceph-mon[35521]: osdmap e30: 4 total, 4 up, 4 in 2024-03-16T10:59:55.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:55 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/4105508442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]: dispatch 2024-03-16T10:59:55.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:55 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]: dispatch 2024-03-16T10:59:55.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:55 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]': finished 2024-03-16T10:59:55.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:55 smithi153 ceph-mon[35521]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T10:59:55.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:55 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:59:55.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:55 smithi078 ceph-mon[32235]: osdmap e30: 4 total, 4 up, 4 in 2024-03-16T10:59:55.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:55 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/4105508442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]: dispatch 2024-03-16T10:59:55.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:55 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]: dispatch 2024-03-16T10:59:55.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:55 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d39555d-45cf-4507-80f8-17103f55bf4a"}]': finished 2024-03-16T10:59:55.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:55 smithi078 ceph-mon[32235]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T10:59:55.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:55 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:59:56.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:56 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/2490174040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:56.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:56 smithi098 ceph-mon[35848]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-16T10:59:56.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:56 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/2490174040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:56.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:56 smithi153 ceph-mon[35521]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-16T10:59:56.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:56 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/2490174040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:59:56.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:56 smithi078 ceph-mon[32235]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-16T10:59:59.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 10:59:58 smithi153 ceph-mon[35521]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-16T10:59:59.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 10:59:58 smithi078 ceph-mon[32235]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-16T10:59:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 10:59:58 smithi098 ceph-mon[35848]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-16T11:00:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:00 smithi098 ceph-mon[35848]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:00:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:00 smithi098 ceph-mon[35848]: overall HEALTH_OK 2024-03-16T11:00:01.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:00 smithi153 ceph-mon[35521]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:00:01.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:00 smithi153 ceph-mon[35521]: overall HEALTH_OK 2024-03-16T11:00:01.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:00 smithi078 ceph-mon[32235]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:00:01.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:00 smithi078 ceph-mon[32235]: overall HEALTH_OK 2024-03-16T11:00:03.243 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:02 smithi098 ceph-mon[35848]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:00:03.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T11:00:03.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:03.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:02 smithi153 ceph-mon[35521]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:00:03.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T11:00:03.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:03.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:02 smithi078 ceph-mon[32235]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:00:03.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T11:00:03.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:04.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:03 smithi098 ceph-mon[35848]: Deploying daemon osd.4 on smithi098 2024-03-16T11:00:04.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:03 smithi153 ceph-mon[35521]: Deploying daemon osd.4 on smithi098 2024-03-16T11:00:04.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:03 smithi078 ceph-mon[32235]: Deploying daemon osd.4 on smithi098 2024-03-16T11:00:05.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:04 smithi098 ceph-mon[35848]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:00:05.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:04 smithi153 ceph-mon[35521]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:00:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:04 smithi078 ceph-mon[32235]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:00:07.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:06 smithi098 ceph-mon[35848]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-16T11:00:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:06 smithi153 ceph-mon[35521]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-16T11:00:07.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:06 smithi078 ceph-mon[32235]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-16T11:00:09.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:08 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:09.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:08 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:09.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:08 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:09.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:08 smithi078 ceph-mon[32235]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T11:00:09.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:08 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:09.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:08 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:09.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:08 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:09.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:08 smithi098 ceph-mon[35848]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T11:00:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:08 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:08 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:08 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:08 smithi153 ceph-mon[35521]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T11:00:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.068+0000 7f3b9cd99700 1 -- 172.21.15.98:0/3775255271 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3ba0062710 con 0x7f3b8c0a2890 2024-03-16T11:00:11.070 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 4 on host 'smithi098' 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b8c0a2890 msgr2=0x7f3b8c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b8c0a2890 0x7f3b8c0a4d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b88005950 tx=0x7f3b880058e0 comp rx=0 tx=0).stop 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 msgr2=0x7f3ba01983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba01983e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3b900098e0 tx=0x7f3b900057c0 comp rx=0 tx=0).stop 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 shutdown_connections 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b8c0a2890 0x7f3b8c0a4d50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ba0103ec0 0x7f3ba019f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba0102320 0x7f3ba01983e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 --2- 172.21.15.98:0/3775255271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba0103520 0x7f3ba0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 >> 172.21.15.98:0/3775255271 conn(0x7f3ba00fd8e0 msgr2=0x7f3ba0106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 shutdown_connections 2024-03-16T11:00:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:00:11.071+0000 7f3ba5092700 1 -- 172.21.15.98:0/3775255271 wait complete. 2024-03-16T11:00:11.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:10 smithi153 ceph-mon[35521]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:11.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:10 smithi078 ceph-mon[32235]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:11.372 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:10 smithi098 ceph-mon[35848]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:11.681 DEBUG:teuthology.orchestra.run.smithi098:osd.4> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.4.service 2024-03-16T11:00:11.683 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_3... 2024-03-16T11:00:11.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T11:00:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:00:11.928 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:00:11 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4[47345]: 2024-03-16T11:00:11.732+0000 7fade6c897c0 -1 osd.4 0 log_to_monitors true 2024-03-16T11:00:12.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:12 smithi098 ceph-mon[35848]: from='osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:00:12.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:12 smithi098 ceph-mon[35848]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:00:12.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:12 smithi153 ceph-mon[35521]: from='osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:00:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:12 smithi153 ceph-mon[35521]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:00:12.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:12.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:12 smithi078 ceph-mon[32235]: from='osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:00:12.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:12 smithi078 ceph-mon[32235]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:00:13.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:13 smithi153 ceph-mon[35521]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:13.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:13 smithi153 ceph-mon[35521]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T11:00:13.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:13 smithi153 ceph-mon[35521]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T11:00:13.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:13.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:13 smithi153 ceph-mon[35521]: from='osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:00:13.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:13 smithi153 ceph-mon[35521]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:00:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:13 smithi078 ceph-mon[32235]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:13 smithi078 ceph-mon[32235]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T11:00:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:13 smithi078 ceph-mon[32235]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T11:00:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:13 smithi078 ceph-mon[32235]: from='osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:00:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:13 smithi078 ceph-mon[32235]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:00:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-mon[35848]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-mon[35848]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T11:00:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-mon[35848]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T11:00:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-mon[35848]: from='osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:00:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-mon[35848]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:00:13.466 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4[47345]: 2024-03-16T11:00:13.156+0000 7faddc2ee700 -1 osd.4 0 waiting for initial osdmap 2024-03-16T11:00:13.466 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:00:13 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4[47345]: 2024-03-16T11:00:13.163+0000 7fadd80e1700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:00:13.798 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T11:00:13.798 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T11:00:13.798 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T11:00:13.799 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T11:00:13.799 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118477 s, 885 MB/s 2024-03-16T11:00:13.799 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T11:00:14.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-03-16T11:00:14.365 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:14 smithi153 ceph-mon[35521]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:00:14.365 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:14 smithi153 ceph-mon[35521]: osdmap e33: 5 total, 4 up, 5 in 2024-03-16T11:00:14.366 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:14 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:14.366 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:14 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:14.366 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:14 smithi153 ceph-mon[35521]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:14.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:14 smithi098 ceph-mon[35848]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:00:14.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:14 smithi098 ceph-mon[35848]: osdmap e33: 5 total, 4 up, 5 in 2024-03-16T11:00:14.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:14 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:14.405 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:14 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:14.405 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:14 smithi098 ceph-mon[35848]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:14.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:14 smithi078 ceph-mon[32235]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:00:14.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:14 smithi078 ceph-mon[32235]: osdmap e33: 5 total, 4 up, 5 in 2024-03-16T11:00:14.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:14 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:14.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:14 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:14.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:14 smithi078 ceph-mon[32235]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:00:14.589 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:00:15.397 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] boot 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi098 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi098 to 8544M 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:15.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:15.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:15.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:15.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:15.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:00:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] boot 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi098 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi098 to 8544M 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:15.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:15.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:00:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:00:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: osd.4 [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] boot 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi098 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi098 to 8544M 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:15.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:16.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 -- 172.21.15.153:0/2902489448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9501003a0 msgr2=0x7ff950100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:16.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 --2- 172.21.15.153:0/2902489448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9501003a0 0x7ff950100800 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff940009990 tx=0x7ff940009ca0 comp rx=0 tx=0).stop 2024-03-16T11:00:16.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 -- 172.21.15.153:0/2902489448 shutdown_connections 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 --2- 172.21.15.153:0/2902489448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff950100d40 0x7ff95010a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 --2- 172.21.15.153:0/2902489448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff950103920 0x7ff950103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 --2- 172.21.15.153:0/2902489448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9501003a0 0x7ff950100800 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 -- 172.21.15.153:0/2902489448 >> 172.21.15.153:0/2902489448 conn(0x7ff9500fbf40 msgr2=0x7ff9500fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 -- 172.21.15.153:0/2902489448 shutdown_connections 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 -- 172.21.15.153:0/2902489448 wait complete. 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 Processor -- start 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.136+0000 7ff9563df700 1 -- start start 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff9563df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9501003a0 0x7ff9501983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff9563df700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff950100d40 0x7ff950198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff9563df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff950103920 0x7ff95019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff9563df700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff95010dd80 con 0x7ff950103920 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff9563df700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff95010dc00 con 0x7ff9501003a0 2024-03-16T11:00:16.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff9563df700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff95010df00 con 0x7ff950100d40 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff950100d40 0x7ff950198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff950100d40 0x7ff950198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54678/0 (socket says 172.21.15.153:54678) 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 -- 172.21.15.153:0/2642758407 learned_addr learned my addr 172.21.15.153:0/2642758407 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94ffff700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9501003a0 0x7ff9501983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff95497c700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff950103920 0x7ff95019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 -- 172.21.15.153:0/2642758407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9501003a0 msgr2=0x7ff9501983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9501003a0 0x7ff9501983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 -- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff950103920 msgr2=0x7ff95019f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff950103920 0x7ff95019f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 -- 172.21.15.153:0/2642758407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9501a0050 con 0x7ff950100d40 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94ffff700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9501003a0 0x7ff9501983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff95497c700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff950103920 0x7ff95019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94f7fe700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff950100d40 0x7ff950198930 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff940005910 tx=0x7ff940009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:00:16.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff94d7fa700 1 -- 172.21.15.153:0/2642758407 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940026070 con 0x7ff950100d40 2024-03-16T11:00:16.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.137+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9501a02e0 con 0x7ff950100d40 2024-03-16T11:00:16.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.138+0000 7ff94d7fa700 1 -- 172.21.15.153:0/2642758407 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff94000fbc0 con 0x7ff950100d40 2024-03-16T11:00:16.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.138+0000 7ff94d7fa700 1 -- 172.21.15.153:0/2642758407 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940017650 con 0x7ff950100d40 2024-03-16T11:00:16.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.138+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9501a0730 con 0x7ff950100d40 2024-03-16T11:00:16.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.139+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff950068f90 con 0x7ff950100d40 2024-03-16T11:00:16.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.140+0000 7ff94d7fa700 1 -- 172.21.15.153:0/2642758407 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff940029070 con 0x7ff950100d40 2024-03-16T11:00:16.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.140+0000 7ff94d7fa700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9380a2890 0x7ff9380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:16.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.140+0000 7ff94d7fa700 1 -- 172.21.15.153:0/2642758407 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7ff9400ce8a0 con 0x7ff950100d40 2024-03-16T11:00:16.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.140+0000 7ff94ffff700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9380a2890 0x7ff9380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:16.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.141+0000 7ff94ffff700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9380a2890 0x7ff9380a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff950069c60 tx=0x7ff944009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:00:16.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.145+0000 7ff94d7fa700 1 -- 172.21.15.153:0/2642758407 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff94005d740 con 0x7ff950100d40 2024-03-16T11:00:16.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:16.350+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff950062710 con 0x7ff9380a2890 2024-03-16T11:00:16.951 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:16 smithi153 ceph-mon[35521]: osdmap e35: 5 total, 5 up, 5 in 2024-03-16T11:00:16.951 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:16 smithi153 ceph-mon[35521]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:16.951 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:00:16.951 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:00:16.951 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:16 smithi098 ceph-mon[35848]: osdmap e35: 5 total, 5 up, 5 in 2024-03-16T11:00:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:16 smithi098 ceph-mon[35848]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:16.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:00:16.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:00:16.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:17.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:16 smithi078 ceph-mon[32235]: osdmap e35: 5 total, 5 up, 5 in 2024-03-16T11:00:17.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:16 smithi078 ceph-mon[32235]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:17.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:00:17.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:00:17.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:17.898 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:17 smithi153 ceph-mon[35521]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:00:17.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:17 smithi098 ceph-mon[35848]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:00:18.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:17 smithi078 ceph-mon[32235]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:00:18.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:18 smithi098 ceph-mon[35848]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:19.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:18 smithi153 ceph-mon[35521]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:19.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:18 smithi078 ceph-mon[32235]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:19.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:19 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/1370499945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]: dispatch 2024-03-16T11:00:19.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:19 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]: dispatch 2024-03-16T11:00:19.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:19 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]': finished 2024-03-16T11:00:19.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:19 smithi098 ceph-mon[35848]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T11:00:19.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:19 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:19.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:19 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/453707274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:00:20.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:19 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/1370499945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]: dispatch 2024-03-16T11:00:20.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:19 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]: dispatch 2024-03-16T11:00:20.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:19 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]': finished 2024-03-16T11:00:20.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:19 smithi153 ceph-mon[35521]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T11:00:20.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:19 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:20.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:19 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/453707274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:00:20.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:19 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/1370499945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]: dispatch 2024-03-16T11:00:20.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:19 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]: dispatch 2024-03-16T11:00:20.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:19 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f"}]': finished 2024-03-16T11:00:20.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:19 smithi078 ceph-mon[32235]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T11:00:20.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:19 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:20.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:19 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/453707274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:00:20.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:20 smithi098 ceph-mon[35848]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:21.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:20 smithi153 ceph-mon[35521]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:21.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:20 smithi078 ceph-mon[32235]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:23.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:22 smithi153 ceph-mon[35521]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:23.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:22 smithi078 ceph-mon[32235]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:22 smithi098 ceph-mon[35848]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:25.012 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:24 smithi153 ceph-mon[35521]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:25.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:24 smithi078 ceph-mon[32235]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:24 smithi098 ceph-mon[35848]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:27.247 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:26 smithi153 ceph-mon[35521]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:27.247 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T11:00:27.247 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:26 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:27.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:26 smithi078 ceph-mon[32235]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:27.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T11:00:27.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:26 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:26 smithi098 ceph-mon[35848]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T11:00:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:26 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:28.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:27 smithi153 ceph-mon[35521]: Deploying daemon osd.5 on smithi153 2024-03-16T11:00:28.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:27 smithi078 ceph-mon[32235]: Deploying daemon osd.5 on smithi153 2024-03-16T11:00:28.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:27 smithi098 ceph-mon[35848]: Deploying daemon osd.5 on smithi153 2024-03-16T11:00:29.156 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:28 smithi153 ceph-mon[35521]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:29.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:28 smithi078 ceph-mon[32235]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:29.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:28 smithi098 ceph-mon[35848]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:31.156 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:30 smithi153 ceph-mon[35521]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:31.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:30 smithi078 ceph-mon[32235]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:30 smithi098 ceph-mon[35848]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:33.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:33.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:33.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:33.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:32 smithi153 ceph-mon[35521]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:33.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:33.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:33.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:33.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:32 smithi078 ceph-mon[32235]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:33.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:33.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:33.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:33.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:32 smithi098 ceph-mon[35848]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:35.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:34 smithi153 ceph-mon[35521]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:35.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.158 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:35.159 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:35.159 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.215 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-03-16T11:00:35.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.214+0000 7ff94d7fa700 1 -- 172.21.15.153:0/2642758407 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff950062710 con 0x7ff9380a2890 2024-03-16T11:00:35.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9380a2890 msgr2=0x7ff9380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:35.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9380a2890 0x7ff9380a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff950069c60 tx=0x7ff944009260 comp rx=0 tx=0).stop 2024-03-16T11:00:35.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff950100d40 msgr2=0x7ff950198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:35.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff950100d40 0x7ff950198930 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff940005910 tx=0x7ff940009580 comp rx=0 tx=0).stop 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 shutdown_connections 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9380a2890 0x7ff9380a4d50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff950103920 0x7ff95019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9501003a0 0x7ff9501983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 --2- 172.21.15.153:0/2642758407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff950100d40 0x7ff950198930 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 >> 172.21.15.153:0/2642758407 conn(0x7ff9500fbf40 msgr2=0x7ff950106a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 shutdown_connections 2024-03-16T11:00:35.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:00:35.217+0000 7ff9563df700 1 -- 172.21.15.153:0/2642758407 wait complete. 2024-03-16T11:00:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:34 smithi078 ceph-mon[32235]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:35.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:34 smithi098 ceph-mon[35848]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:35.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:35.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:35.873 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.5.service 2024-03-16T11:00:35.915 INFO:tasks.cephadm:Deploying osd.6 on smithi078 with /dev/vg_nvme/lv_2... 2024-03-16T11:00:35.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T11:00:36.160 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:00:36.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:36.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:36.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:36.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:36 smithi078 ceph-mon[32235]: from='osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:00:36.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:36 smithi078 ceph-mon[32235]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:00:36.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:36 smithi078 ceph-mon[32235]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:36.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:36 smithi098 ceph-mon[35848]: from='osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:00:36.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:36 smithi098 ceph-mon[35848]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:00:36.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:36 smithi098 ceph-mon[35848]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:36.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:36.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:36.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:36.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:36 smithi153 ceph-mon[35521]: from='osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:00:36.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:36 smithi153 ceph-mon[35521]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:00:36.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:36 smithi153 ceph-mon[35521]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:37 smithi078 ceph-mon[32235]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T11:00:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:37 smithi078 ceph-mon[32235]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T11:00:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:37 smithi078 ceph-mon[32235]: from='osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:00:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:37 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:37.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:37 smithi078 ceph-mon[32235]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:00:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:37 smithi098 ceph-mon[35848]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T11:00:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:37 smithi098 ceph-mon[35848]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T11:00:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:37 smithi098 ceph-mon[35848]: from='osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:00:37.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:37 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:37.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:37 smithi098 ceph-mon[35848]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:00:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:37 smithi153 ceph-mon[35521]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T11:00:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:37 smithi153 ceph-mon[35521]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T11:00:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:37 smithi153 ceph-mon[35521]: from='osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:00:37.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:37 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:37.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:37 smithi153 ceph-mon[35521]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:00:38.204 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T11:00:38.204 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T11:00:38.204 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-16T11:00:38.204 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-16T11:00:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122291 s, 857 MB/s 2024-03-16T11:00:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-16T11:00:38.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-mon[35521]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:00:38.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-mon[35521]: osdmap e38: 6 total, 5 up, 6 in 2024-03-16T11:00:38.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:38.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:38.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-mon[35521]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:38.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-mon[35521]: from='osd.5 ' entity='osd.5' 2024-03-16T11:00:38.494 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:00:38.130+0000 7fe3e5246700 -1 osd.5 0 waiting for initial osdmap 2024-03-16T11:00:38.494 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:00:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:00:38.207+0000 7fe3dd031700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:00:38.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:38 smithi078 ceph-mon[32235]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:00:38.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:38 smithi078 ceph-mon[32235]: osdmap e38: 6 total, 5 up, 6 in 2024-03-16T11:00:38.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:38.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:38.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:38 smithi078 ceph-mon[32235]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:38.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:38 smithi078 ceph-mon[32235]: from='osd.5 ' entity='osd.5' 2024-03-16T11:00:38.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:38 smithi098 ceph-mon[35848]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:00:38.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:38 smithi098 ceph-mon[35848]: osdmap e38: 6 total, 5 up, 6 in 2024-03-16T11:00:38.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:38.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:38.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:38 smithi098 ceph-mon[35848]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:38.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:38 smithi098 ceph-mon[35848]: from='osd.5 ' entity='osd.5' 2024-03-16T11:00:38.891 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2024-03-16T11:00:39.132 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:00:39.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:00:39.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:00:39.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: osdmap e39: 6 total, 5 up, 6 in 2024-03-16T11:00:39.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:39.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi153 2024-03-16T11:00:39.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi153 to 8544M 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] boot 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T11:00:39.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: osdmap e39: 6 total, 5 up, 6 in 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi153 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi153 to 8544M 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:39.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] boot 2024-03-16T11:00:39.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T11:00:39.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:39.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: osdmap e39: 6 total, 5 up, 6 in 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi153 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi153 to 8544M 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: osd.5 [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] boot 2024-03-16T11:00:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T11:00:39.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:00:40.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:40 smithi078 ceph-mon[32235]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:40.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:40 smithi078 ceph-mon[32235]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T11:00:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:40 smithi153 ceph-mon[35521]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:40 smithi153 ceph-mon[35521]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T11:00:40.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:40 smithi098 ceph-mon[35848]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:00:40.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:40 smithi098 ceph-mon[35848]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T11:00:40.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.773+0000 7f3fb5589700 1 -- 172.21.15.78:0/3175352307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 msgr2=0x7f3fb0102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:40.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.773+0000 7f3fb5589700 1 --2- 172.21.15.78:0/3175352307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb0102860 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40099c0 tx=0x7f3fa4009cd0 comp rx=0 tx=0).stop 2024-03-16T11:00:40.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 -- 172.21.15.78:0/3175352307 shutdown_connections 2024-03-16T11:00:40.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 --2- 172.21.15.78:0/3175352307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102da0 0x7f3fb0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:40.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 --2- 172.21.15.78:0/3175352307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb0101200 0x7f3fb0101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:40.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 --2- 172.21.15.78:0/3175352307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb0102860 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:40.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 -- 172.21.15.78:0/3175352307 >> 172.21.15.78:0/3175352307 conn(0x7f3fb00fc7a0 msgr2=0x7f3fb00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:00:40.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 -- 172.21.15.78:0/3175352307 shutdown_connections 2024-03-16T11:00:40.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 -- 172.21.15.78:0/3175352307 wait complete. 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.774+0000 7f3fb5589700 1 Processor -- start 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fb5589700 1 -- start start 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fb5589700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0101200 0x7f3fb01973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fb5589700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb01978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fb5589700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb0102da0 0x7f3fb019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fb5589700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb010cd10 con 0x7f3fb0102400 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fb5589700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb010cb90 con 0x7f3fb0102da0 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fb5589700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb010ce90 con 0x7f3fb0101200 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb01978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb01978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54398/0 (socket says 172.21.15.78:54398) 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 -- 172.21.15.78:0/900310367 learned_addr learned my addr 172.21.15.78:0/900310367 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:00:40.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3faf7fe700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb0102da0 0x7f3fb019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3faeffd700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0101200 0x7f3fb01973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 -- 172.21.15.78:0/900310367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0101200 msgr2=0x7f3fb01973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0101200 0x7f3fb01973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 -- 172.21.15.78:0/900310367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb0102da0 msgr2=0x7f3fb019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb0102da0 0x7f3fb019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.775+0000 7f3fae7fc700 1 -- 172.21.15.78:0/900310367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb019f010 con 0x7f3fb0102400 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.776+0000 7f3fae7fc700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb01978f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4000c00 tx=0x7f3fa40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.776+0000 7f3f97fff700 1 -- 172.21.15.78:0/900310367 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4027070 con 0x7f3fb0102400 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.776+0000 7f3f97fff700 1 -- 172.21.15.78:0/900310367 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fa4004050 con 0x7f3fb0102400 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.776+0000 7f3f97fff700 1 -- 172.21.15.78:0/900310367 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4022d70 con 0x7f3fb0102400 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.776+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb0069f80 con 0x7f3fb0102400 2024-03-16T11:00:40.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.776+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb006a4e0 con 0x7f3fb0102400 2024-03-16T11:00:40.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.777+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fb0068f90 con 0x7f3fb0102400 2024-03-16T11:00:40.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.778+0000 7f3f97fff700 1 -- 172.21.15.78:0/900310367 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3fa402a070 con 0x7f3fb0102400 2024-03-16T11:00:40.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.779+0000 7f3f97fff700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3f980a2890 0x7f3f980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:00:40.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.779+0000 7f3faeffd700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3f980a2890 0x7f3f980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:00:40.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.779+0000 7f3f97fff700 1 -- 172.21.15.78:0/900310367 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f3fa40cf160 con 0x7f3fb0102400 2024-03-16T11:00:40.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.779+0000 7f3faeffd700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3f980a2890 0x7f3f980a4d50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0102260 tx=0x7f3fa800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:00:40.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.782+0000 7f3f97fff700 1 -- 172.21.15.78:0/900310367 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3fa4031020 con 0x7f3fb0102400 2024-03-16T11:00:40.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:00:40.986+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3fb006a740 con 0x7f3f980a2890 2024-03-16T11:00:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:41 smithi153 ceph-mon[35521]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:00:41.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:00:41.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:00:41.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:41.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:41 smithi153 ceph-mon[35521]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T11:00:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:41 smithi078 ceph-mon[32235]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:00:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:00:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:00:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:41 smithi078 ceph-mon[32235]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T11:00:41.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:41 smithi098 ceph-mon[35848]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:00:41.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:00:41.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:00:41.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:41.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:41 smithi098 ceph-mon[35848]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T11:00:42.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:42 smithi078 ceph-mon[32235]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:42.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:42 smithi153 ceph-mon[35521]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:42.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:42 smithi098 ceph-mon[35848]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:43 smithi153 ceph-mon[35521]: osdmap e43: 6 total, 6 up, 6 in 2024-03-16T11:00:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:43 smithi078 ceph-mon[32235]: osdmap e43: 6 total, 6 up, 6 in 2024-03-16T11:00:43.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:43 smithi098 ceph-mon[35848]: osdmap e43: 6 total, 6 up, 6 in 2024-03-16T11:00:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:44 smithi153 ceph-mon[35521]: osdmap e44: 6 total, 6 up, 6 in 2024-03-16T11:00:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1317478643' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4ef5595-a5ca-4f0e-82a7-f8322daab5e6"}]: dispatch 2024-03-16T11:00:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1317478643' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4ef5595-a5ca-4f0e-82a7-f8322daab5e6"}]': finished 2024-03-16T11:00:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:44 smithi153 ceph-mon[35521]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T11:00:44.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:00:44.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:44 smithi153 ceph-mon[35521]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-03-16T11:00:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:44 smithi078 ceph-mon[32235]: osdmap e44: 6 total, 6 up, 6 in 2024-03-16T11:00:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1317478643' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4ef5595-a5ca-4f0e-82a7-f8322daab5e6"}]: dispatch 2024-03-16T11:00:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1317478643' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4ef5595-a5ca-4f0e-82a7-f8322daab5e6"}]': finished 2024-03-16T11:00:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:44 smithi078 ceph-mon[32235]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T11:00:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:00:44.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:44 smithi078 ceph-mon[32235]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-03-16T11:00:44.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:44 smithi098 ceph-mon[35848]: osdmap e44: 6 total, 6 up, 6 in 2024-03-16T11:00:44.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1317478643' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4ef5595-a5ca-4f0e-82a7-f8322daab5e6"}]: dispatch 2024-03-16T11:00:44.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1317478643' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4ef5595-a5ca-4f0e-82a7-f8322daab5e6"}]': finished 2024-03-16T11:00:44.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:44 smithi098 ceph-mon[35848]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T11:00:44.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:00:44.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:44 smithi098 ceph-mon[35848]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-03-16T11:00:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:45 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2211206930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:00:45.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:45 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2211206930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:00:45.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:45 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2211206930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:00:46.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:46 smithi153 ceph-mon[35521]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T11:00:46.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:46 smithi078 ceph-mon[32235]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T11:00:46.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:46 smithi098 ceph-mon[35848]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T11:00:49.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:48 smithi078 ceph-mon[32235]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T11:00:49.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:48 smithi153 ceph-mon[35521]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T11:00:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:48 smithi098 ceph-mon[35848]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T11:00:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:50 smithi153 ceph-mon[35521]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-16T11:00:51.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:50 smithi078 ceph-mon[32235]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-16T11:00:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:50 smithi098 ceph-mon[35848]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-16T11:00:52.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T11:00:52.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:52.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T11:00:52.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:52.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T11:00:52.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:00:53.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:52 smithi078 ceph-mon[32235]: Deploying daemon osd.6 on smithi078 2024-03-16T11:00:53.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:52 smithi078 ceph-mon[32235]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:52 smithi153 ceph-mon[35521]: Deploying daemon osd.6 on smithi078 2024-03-16T11:00:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:52 smithi153 ceph-mon[35521]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:52 smithi098 ceph-mon[35848]: Deploying daemon osd.6 on smithi078 2024-03-16T11:00:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:52 smithi098 ceph-mon[35848]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:54 smithi078 ceph-mon[32235]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:55.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:54 smithi153 ceph-mon[35521]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:55.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:54 smithi098 ceph-mon[35848]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:57.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:56 smithi153 ceph-mon[35521]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:56 smithi078 ceph-mon[32235]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:56 smithi098 ceph-mon[35848]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:00:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:00:59.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:00:58 smithi153 ceph-mon[35521]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:59.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:00:58 smithi078 ceph-mon[32235]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:00:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:00:58 smithi098 ceph-mon[35848]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:01:00.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.805+0000 7f3f97fff700 1 -- 172.21.15.78:0/900310367 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3fb006a740 con 0x7f3f980a2890 2024-03-16T11:01:00.806 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 6 on host 'smithi078' 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3f980a2890 msgr2=0x7f3f980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3f980a2890 0x7f3f980a4d50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0102260 tx=0x7f3fa800d040 comp rx=0 tx=0).stop 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 msgr2=0x7f3fb01978f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb01978f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4000c00 tx=0x7f3fa40057c0 comp rx=0 tx=0).stop 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 shutdown_connections 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3f980a2890 0x7f3f980a4d50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0101200 0x7f3fb01973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fb0102da0 0x7f3fb019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 --2- 172.21.15.78:0/900310367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0102400 0x7f3fb01978f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 >> 172.21.15.78:0/900310367 conn(0x7f3fb00fc7a0 msgr2=0x7f3fb0105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 shutdown_connections 2024-03-16T11:01:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:00.807+0000 7f3fb5589700 1 -- 172.21.15.78:0/900310367 wait complete. 2024-03-16T11:01:01.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:01.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:01.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:01:01.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T11:01:01.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:00 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:00 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:01.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:00 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:01.527 DEBUG:teuthology.orchestra.run.smithi078:osd.6> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.6.service 2024-03-16T11:01:01.530 INFO:tasks.cephadm:Deploying osd.7 on smithi098 with /dev/vg_nvme/lv_2... 2024-03-16T11:01:01.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T11:01:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:01:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:01 smithi078 ceph-mon[32235]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T11:01:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:01 smithi078 ceph-mon[32235]: osdmap e46: 7 total, 6 up, 7 in 2024-03-16T11:01:02.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:01 smithi078 ceph-mon[32235]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T11:01:02.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:02.060 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:01:01 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:01:01.888+0000 7f618e669700 -1 osd.6 0 waiting for initial osdmap 2024-03-16T11:01:02.060 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:01:01 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:01:01.900+0000 7f618a45c700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:01:02.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:01 smithi098 ceph-mon[35848]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T11:01:02.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:01 smithi098 ceph-mon[35848]: osdmap e46: 7 total, 6 up, 7 in 2024-03-16T11:01:02.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:01 smithi098 ceph-mon[35848]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T11:01:02.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:01 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:02.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:01 smithi153 ceph-mon[35521]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T11:01:02.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:01 smithi153 ceph-mon[35521]: osdmap e46: 7 total, 6 up, 7 in 2024-03-16T11:01:02.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:01 smithi153 ceph-mon[35521]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T11:01:02.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:03.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:02 smithi078 ceph-mon[32235]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T11:01:03.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:02 smithi078 ceph-mon[32235]: osdmap e47: 7 total, 6 up, 7 in 2024-03-16T11:01:03.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:03.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:03.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:02 smithi078 ceph-mon[32235]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:01:03.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:02 smithi098 ceph-mon[35848]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T11:01:03.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:02 smithi098 ceph-mon[35848]: osdmap e47: 7 total, 6 up, 7 in 2024-03-16T11:01:03.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:03.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:03.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:02 smithi098 ceph-mon[35848]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:01:03.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:02 smithi153 ceph-mon[35521]: from='osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T11:01:03.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:02 smithi153 ceph-mon[35521]: osdmap e47: 7 total, 6 up, 7 in 2024-03-16T11:01:03.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:03.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:03.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:02 smithi153 ceph-mon[35521]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:01:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T11:01:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T11:01:03.799 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T11:01:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T11:01:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132901 s, 789 MB/s 2024-03-16T11:01:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T11:01:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:03 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:01:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:03 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:01:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:03 smithi078 ceph-mon[32235]: osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] boot 2024-03-16T11:01:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:03 smithi078 ceph-mon[32235]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T11:01:04.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:04.091 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:03 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:01:04.091 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:03 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:01:04.091 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:04.091 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:03 smithi098 ceph-mon[35848]: osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] boot 2024-03-16T11:01:04.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:03 smithi098 ceph-mon[35848]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T11:01:04.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:04.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:03 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:01:04.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:03 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:01:04.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:04.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:03 smithi153 ceph-mon[35521]: osd.6 [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] boot 2024-03-16T11:01:04.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:03 smithi153 ceph-mon[35521]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T11:01:04.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:01:04.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-03-16T11:01:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:01:04.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: osdmap e49: 7 total, 7 up, 7 in 2024-03-16T11:01:04.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:04.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:04 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: osdmap e49: 7 total, 7 up, 7 in 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:04 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: osdmap e49: 7 total, 7 up, 7 in 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:05.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:04 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 -- 172.21.15.98:0/1635479573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c076300 msgr2=0x7fbb4c074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 --2- 172.21.15.98:0/1635479573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c076300 0x7fbb4c074760 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009990 tx=0x7fbb48009ca0 comp rx=0 tx=0).stop 2024-03-16T11:01:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 -- 172.21.15.98:0/1635479573 shutdown_connections 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 --2- 172.21.15.98:0/1635479573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb4c10e8e0 0x7fbb4c110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 --2- 172.21.15.98:0/1635479573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c076300 0x7fbb4c074760 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 --2- 172.21.15.98:0/1635479573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb4c074d30 0x7fbb4c0751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 -- 172.21.15.98:0/1635479573 >> 172.21.15.98:0/1635479573 conn(0x7fbb4c0fd8e0 msgr2=0x7fbb4c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 -- 172.21.15.98:0/1635479573 shutdown_connections 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.013+0000 7fbb54060700 1 -- 172.21.15.98:0/1635479573 wait complete. 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 Processor -- start 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 -- start start 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb4c074d30 0x7fbb4c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb4c076300 0x7fbb4c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:06.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c10e8e0 0x7fbb4c1a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c114490 con 0x7fbb4c074d30 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c114310 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb54060700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c114610 con 0x7fbb4c076300 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c10e8e0 0x7fbb4c1a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c10e8e0 0x7fbb4c1a6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47160/0 (socket says 172.21.15.98:47160) 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 -- 172.21.15.98:0/1902047033 learned_addr learned my addr 172.21.15.98:0/1902047033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb515fb700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb4c076300 0x7fbb4c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb51dfc700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb4c074d30 0x7fbb4c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 -- 172.21.15.98:0/1902047033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb4c076300 msgr2=0x7fbb4c19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb4c076300 0x7fbb4c19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 -- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb4c074d30 msgr2=0x7fbb4c19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb4c074d30 0x7fbb4c19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 -- 172.21.15.98:0/1902047033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4c1a67a0 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb51dfc700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb4c074d30 0x7fbb4c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.014+0000 7fbb525fd700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c10e8e0 0x7fbb4c1a6100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400b970 tx=0x7fbb4400bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.015+0000 7fbb42ffd700 1 -- 172.21.15.98:0/1902047033 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44019070 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.015+0000 7fbb42ffd700 1 -- 172.21.15.98:0/1902047033 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb44014440 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.015+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb4c1a6a90 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.015+0000 7fbb42ffd700 1 -- 172.21.15.98:0/1902047033 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400ce60 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.015+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb4c1a6f80 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.016+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb4c068f90 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.017+0000 7fbb42ffd700 1 -- 172.21.15.98:0/1902047033 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbb4401c070 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.017+0000 7fbb42ffd700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbb380a2890 0x7fbb380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.017+0000 7fbb42ffd700 1 -- 172.21.15.98:0/1902047033 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4965+0+0 (secure 0 0 0) 0x7fbb440cd630 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.017+0000 7fbb51dfc700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbb380a2890 0x7fbb380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.018+0000 7fbb51dfc700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbb380a2890 0x7fbb380a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbb48000c00 tx=0x7fbb48005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:01:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.021+0000 7fbb42ffd700 1 -- 172.21.15.98:0/1902047033 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb4409b3e0 con 0x7fbb4c10e8e0 2024-03-16T11:01:06.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:05 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi078 2024-03-16T11:01:06.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:05 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi078 to 5696M 2024-03-16T11:01:06.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:06.226+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbb4c062710 con 0x7fbb380a2890 2024-03-16T11:01:06.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:05 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi078 2024-03-16T11:01:06.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:05 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi078 to 5696M 2024-03-16T11:01:06.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:05 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi078 2024-03-16T11:01:06.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:05 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi078 to 5696M 2024-03-16T11:01:07.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:06 smithi098 ceph-mon[35848]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:07.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:07.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:07.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:07.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:06 smithi078 ceph-mon[32235]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:07.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:07.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:07.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:06 smithi153 ceph-mon[35521]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:08.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:07 smithi098 ceph-mon[35848]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:08.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:07 smithi153 ceph-mon[35521]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:07 smithi078 ceph-mon[32235]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:09.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:08 smithi098 ceph-mon[35848]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:09.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:08 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/560196666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]: dispatch 2024-03-16T11:01:09.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:08 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]: dispatch 2024-03-16T11:01:09.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:08 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]': finished 2024-03-16T11:01:09.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:08 smithi098 ceph-mon[35848]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T11:01:09.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:08 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:08 smithi153 ceph-mon[35521]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:08 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/560196666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]: dispatch 2024-03-16T11:01:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:08 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]: dispatch 2024-03-16T11:01:09.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:08 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]': finished 2024-03-16T11:01:09.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:08 smithi153 ceph-mon[35521]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T11:01:09.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:08 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:08 smithi078 ceph-mon[32235]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:08 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/560196666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]: dispatch 2024-03-16T11:01:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:08 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]: dispatch 2024-03-16T11:01:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:08 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db466cbf-77b8-4cc4-95ca-a49e51ba27e6"}]': finished 2024-03-16T11:01:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:08 smithi078 ceph-mon[32235]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T11:01:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:08 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:10.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:09 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/3112344196' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:10.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:09 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/3112344196' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:10.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:09 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/3112344196' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:11.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:10 smithi153 ceph-mon[35521]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:11.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:10 smithi098 ceph-mon[35848]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:11.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:10 smithi078 ceph-mon[32235]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:13.016 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:12 smithi098 ceph-mon[35848]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:12 smithi153 ceph-mon[35521]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:13.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:12 smithi078 ceph-mon[32235]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:15.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:14 smithi153 ceph-mon[35521]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:15.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:14 smithi098 ceph-mon[35848]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:15.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:14 smithi078 ceph-mon[32235]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:17.086 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:16 smithi098 ceph-mon[35848]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:17.086 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T11:01:17.087 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:17.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:16 smithi153 ceph-mon[35521]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:17.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T11:01:17.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:17.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:16 smithi078 ceph-mon[32235]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:17.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T11:01:17.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:18.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:17 smithi153 ceph-mon[35521]: Deploying daemon osd.7 on smithi098 2024-03-16T11:01:18.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:17 smithi078 ceph-mon[32235]: Deploying daemon osd.7 on smithi098 2024-03-16T11:01:18.358 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:17 smithi098 ceph-mon[35848]: Deploying daemon osd.7 on smithi098 2024-03-16T11:01:19.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:18 smithi098 ceph-mon[35848]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:19.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:18 smithi153 ceph-mon[35521]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:19.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:18 smithi078 ceph-mon[32235]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:21.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:20 smithi098 ceph-mon[35848]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:21.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:20 smithi153 ceph-mon[35521]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:21.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:20 smithi078 ceph-mon[32235]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:22.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:22.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:22.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:22.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:22.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:22.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:22.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:23.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:23 smithi153 ceph-mon[35521]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:23.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:23 smithi078 ceph-mon[32235]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:23.358 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:23 smithi098 ceph-mon[35848]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:25.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.038+0000 7fbb42ffd700 1 -- 172.21.15.98:0/1902047033 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbb4c062710 con 0x7fbb380a2890 2024-03-16T11:01:25.041 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 7 on host 'smithi098' 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.040+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbb380a2890 msgr2=0x7fbb380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.040+0000 7fbb54060700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbb380a2890 0x7fbb380a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbb48000c00 tx=0x7fbb48005fb0 comp rx=0 tx=0).stop 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.040+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c10e8e0 msgr2=0x7fbb4c1a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.040+0000 7fbb54060700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c10e8e0 0x7fbb4c1a6100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400b970 tx=0x7fbb4400bd30 comp rx=0 tx=0).stop 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 shutdown_connections 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbb380a2890 0x7fbb380a4d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb4c10e8e0 0x7fbb4c1a6100 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:25.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb4c074d30 0x7fbb4c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:25.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 --2- 172.21.15.98:0/1902047033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb4c076300 0x7fbb4c19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:25.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 >> 172.21.15.98:0/1902047033 conn(0x7fbb4c0fd8e0 msgr2=0x7fbb4c101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:01:25.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 shutdown_connections 2024-03-16T11:01:25.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:01:25.041+0000 7fbb54060700 1 -- 172.21.15.98:0/1902047033 wait complete. 2024-03-16T11:01:25.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:25.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:25.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:25.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:25.645 DEBUG:teuthology.orchestra.run.smithi098:osd.7> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.7.service 2024-03-16T11:01:25.648 INFO:tasks.cephadm:Deploying osd.8 on smithi153 with /dev/vg_nvme/lv_2... 2024-03-16T11:01:25.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T11:01:25.897 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:01:26.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:26 smithi153 ceph-mon[35521]: from='osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:01:26.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:26 smithi153 ceph-mon[35521]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:01:26.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:26 smithi098 ceph-mon[35848]: from='osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:01:26.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:26 smithi098 ceph-mon[35848]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:01:26.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:26 smithi078 ceph-mon[32235]: from='osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:01:26.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:26 smithi078 ceph-mon[32235]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:01:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:27 smithi153 ceph-mon[35521]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:27 smithi153 ceph-mon[35521]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T11:01:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:27 smithi153 ceph-mon[35521]: from='osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:01:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:27 smithi153 ceph-mon[35521]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T11:01:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:27 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:27 smithi153 ceph-mon[35521]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:01:27.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-mon[35848]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:27.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-mon[35848]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T11:01:27.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-mon[35848]: from='osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:01:27.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-mon[35848]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T11:01:27.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:27.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-mon[35848]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:01:27.465 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:01:27.159+0000 7fa1180d9700 -1 osd.7 0 waiting for initial osdmap 2024-03-16T11:01:27.466 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:01:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:01:27.170+0000 7fa113ecc700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:01:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:27 smithi078 ceph-mon[32235]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:27 smithi078 ceph-mon[32235]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T11:01:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:27 smithi078 ceph-mon[32235]: from='osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:01:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:27 smithi078 ceph-mon[32235]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T11:01:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:27 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:27.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:27 smithi078 ceph-mon[32235]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:01:27.887 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T11:01:27.887 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T11:01:27.887 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T11:01:27.887 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T11:01:27.887 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011351 s, 924 MB/s 2024-03-16T11:01:27.888 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T11:01:28.414 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:28 smithi098 ceph-mon[35848]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:01:28.414 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:28 smithi098 ceph-mon[35848]: osdmap e52: 8 total, 7 up, 8 in 2024-03-16T11:01:28.414 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:28.414 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:28 smithi098 ceph-mon[35848]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:28.414 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:28 smithi098 ceph-mon[35848]: osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] boot 2024-03-16T11:01:28.415 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:28 smithi098 ceph-mon[35848]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T11:01:28.415 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:28.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:28 smithi153 ceph-mon[35521]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:01:28.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:28 smithi153 ceph-mon[35521]: osdmap e52: 8 total, 7 up, 8 in 2024-03-16T11:01:28.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:28.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:28 smithi153 ceph-mon[35521]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:28.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:28 smithi153 ceph-mon[35521]: osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] boot 2024-03-16T11:01:28.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:28 smithi153 ceph-mon[35521]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T11:01:28.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:28.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-03-16T11:01:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:28 smithi078 ceph-mon[32235]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:01:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:28 smithi078 ceph-mon[32235]: osdmap e52: 8 total, 7 up, 8 in 2024-03-16T11:01:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:28 smithi078 ceph-mon[32235]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:01:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:28 smithi078 ceph-mon[32235]: osd.7 [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] boot 2024-03-16T11:01:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:28 smithi078 ceph-mon[32235]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T11:01:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:01:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:01:29.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:01:29.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:01:29.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T11:01:29.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi098 2024-03-16T11:01:29.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi098 to 5696M 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:29.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:29 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi098 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi098 to 5696M 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:29.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:29.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:29 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi098 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:29.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi098 to 5696M 2024-03-16T11:01:29.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:29.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:29.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:29.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:29 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 -- 172.21.15.153:0/733775924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 msgr2=0x7fc958102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 --2- 172.21.15.153:0/733775924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958102860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc948009990 tx=0x7fc948009ca0 comp rx=0 tx=0).stop 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 -- 172.21.15.153:0/733775924 shutdown_connections 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 --2- 172.21.15.153:0/733775924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc958102da0 0x7fc958109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 --2- 172.21.15.153:0/733775924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc958101200 0x7fc958101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 --2- 172.21.15.153:0/733775924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958102860 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 -- 172.21.15.153:0/733775924 >> 172.21.15.153:0/733775924 conn(0x7fc9580fc7c0 msgr2=0x7fc9580febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 -- 172.21.15.153:0/733775924 shutdown_connections 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 -- 172.21.15.153:0/733775924 wait complete. 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 Processor -- start 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 -- start start 2024-03-16T11:01:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc958101200 0x7fc958198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.249+0000 7fc960072700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc958102da0 0x7fc95819f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc960072700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc95810cc70 con 0x7fc958101200 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc960072700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc95810caf0 con 0x7fc958102da0 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc960072700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc95810cdf0 con 0x7fc958102400 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48238/0 (socket says 172.21.15.153:48238) 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 -- 172.21.15.153:0/1396098512 learned_addr learned my addr 172.21.15.153:0/1396098512 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95de0e700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc958101200 0x7fc958198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95e60f700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc958102da0 0x7fc95819f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 -- 172.21.15.153:0/1396098512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc958102da0 msgr2=0x7fc95819f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc958102da0 0x7fc95819f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:30.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 -- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc958101200 msgr2=0x7fc958198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc958101200 0x7fc958198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 -- 172.21.15.153:0/1396098512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9581a0000 con 0x7fc958102400 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95de0e700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc958101200 0x7fc958198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc95d60d700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958198950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc948000c00 tx=0x7fc948003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc94effd700 1 -- 172.21.15.153:0/1396098512 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948027070 con 0x7fc958102400 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc94effd700 1 -- 172.21.15.153:0/1396098512 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc948004080 con 0x7fc958102400 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc94effd700 1 -- 172.21.15.153:0/1396098512 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948022e30 con 0x7fc958102400 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9581a0290 con 0x7fc958102400 2024-03-16T11:01:30.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.250+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9581a0690 con 0x7fc958102400 2024-03-16T11:01:30.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.252+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc958068fc0 con 0x7fc958102400 2024-03-16T11:01:30.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.253+0000 7fc94effd700 1 -- 172.21.15.153:0/1396098512 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc94802a070 con 0x7fc958102400 2024-03-16T11:01:30.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.253+0000 7fc94effd700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc9440a2890 0x7fc9440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:30.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.253+0000 7fc94effd700 1 -- 172.21.15.153:0/1396098512 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5426+0+0 (secure 0 0 0) 0x7fc9480cf2f0 con 0x7fc958102400 2024-03-16T11:01:30.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.254+0000 7fc95de0e700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc9440a2890 0x7fc9440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:30.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.254+0000 7fc95de0e700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc9440a2890 0x7fc9440a4d50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc954005950 tx=0x7fc9540058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:01:30.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.257+0000 7fc94effd700 1 -- 172.21.15.153:0/1396098512 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc958068fc0 con 0x7fc958102400 2024-03-16T11:01:30.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:30.463+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc9580fe980 con 0x7fc9440a2890 2024-03-16T11:01:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:30 smithi153 ceph-mon[35521]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T11:01:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:30 smithi153 ceph-mon[35521]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:30.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:30 smithi098 ceph-mon[35848]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T11:01:30.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:30 smithi098 ceph-mon[35848]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:30.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:30 smithi078 ceph-mon[32235]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T11:01:30.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:30 smithi078 ceph-mon[32235]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:31.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:31 smithi153 ceph-mon[35521]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T11:01:31.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:31 smithi153 ceph-mon[35521]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:31.692 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:31.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:31.693 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:31.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:31 smithi098 ceph-mon[35848]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T11:01:31.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:31 smithi098 ceph-mon[35848]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:31.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:31.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:31.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:31 smithi078 ceph-mon[32235]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T11:01:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:31 smithi078 ceph-mon[32235]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:31.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:32.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:32 smithi153 ceph-mon[35521]: osdmap e57: 8 total, 8 up, 8 in 2024-03-16T11:01:32.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:32 smithi153 ceph-mon[35521]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:32.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:32 smithi098 ceph-mon[35848]: osdmap e57: 8 total, 8 up, 8 in 2024-03-16T11:01:32.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:32 smithi098 ceph-mon[35848]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:32 smithi078 ceph-mon[32235]: osdmap e57: 8 total, 8 up, 8 in 2024-03-16T11:01:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:32 smithi078 ceph-mon[32235]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:33.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:33 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/3088383125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]: dispatch 2024-03-16T11:01:33.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:33 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]: dispatch 2024-03-16T11:01:33.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:33 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]': finished 2024-03-16T11:01:33.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:33 smithi098 ceph-mon[35848]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T11:01:33.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:33 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:33.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:33 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/3088383125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]: dispatch 2024-03-16T11:01:33.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:33 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]: dispatch 2024-03-16T11:01:33.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:33 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]': finished 2024-03-16T11:01:33.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:33 smithi153 ceph-mon[35521]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T11:01:33.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:33 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:33.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:33 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/3088383125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]: dispatch 2024-03-16T11:01:33.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:33 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]: dispatch 2024-03-16T11:01:33.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:33 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e063f0ca-e609-4d53-8a22-f6ce8f3dae43"}]': finished 2024-03-16T11:01:33.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:33 smithi078 ceph-mon[32235]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T11:01:33.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:33 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:34.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:34 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/1944310730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:34.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:34 smithi153 ceph-mon[35521]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:34.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:34 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/1944310730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:34.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:34 smithi078 ceph-mon[32235]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:34.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:34 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/1944310730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:34.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:34 smithi098 ceph-mon[35848]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:37.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:36 smithi153 ceph-mon[35521]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:36 smithi078 ceph-mon[32235]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:36 smithi098 ceph-mon[35848]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:39.116 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:38 smithi153 ceph-mon[35521]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:39.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:38 smithi078 ceph-mon[32235]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:38 smithi098 ceph-mon[35848]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:40 smithi153 ceph-mon[35521]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:41.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:40 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T11:01:41.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:40 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:40 smithi078 ceph-mon[32235]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T11:01:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:40 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:40 smithi098 ceph-mon[35848]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:41.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:40 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T11:01:41.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:40 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:42.128 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:41 smithi153 ceph-mon[35521]: Deploying daemon osd.8 on smithi153 2024-03-16T11:01:42.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:41 smithi078 ceph-mon[32235]: Deploying daemon osd.8 on smithi153 2024-03-16T11:01:42.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:42 smithi098 ceph-mon[35848]: Deploying daemon osd.8 on smithi153 2024-03-16T11:01:43.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:43 smithi153 ceph-mon[35521]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:43.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:42 smithi078 ceph-mon[32235]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:43.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:43 smithi098 ceph-mon[35848]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:45.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:45 smithi153 ceph-mon[35521]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:45 smithi078 ceph-mon[32235]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:45 smithi098 ceph-mon[35848]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:47 smithi153 ceph-mon[35521]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:47.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:47 smithi098 ceph-mon[35848]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:47.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:47 smithi078 ceph-mon[32235]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:47.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:47.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:47.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:49 smithi098 ceph-mon[35848]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:49 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.488 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:49 smithi153 ceph-mon[35521]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:49.488 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.488 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.488 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:49.488 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:49.489 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:49 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:49 smithi078 ceph-mon[32235]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:49.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:49.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:49.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:49 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:49.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.832+0000 7fc94effd700 1 -- 172.21.15.153:0/1396098512 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc9580fe980 con 0x7fc9440a2890 2024-03-16T11:01:49.834 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 8 on host 'smithi153' 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc9440a2890 msgr2=0x7fc9440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc9440a2890 0x7fc9440a4d50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc954005950 tx=0x7fc9540058e0 comp rx=0 tx=0).stop 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 msgr2=0x7fc958198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958198950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc948000c00 tx=0x7fc948003680 comp rx=0 tx=0).stop 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 shutdown_connections 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc9440a2890 0x7fc9440a4d50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc958102da0 0x7fc95819f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc958101200 0x7fc958198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 --2- 172.21.15.153:0/1396098512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc958102400 0x7fc958198950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:49.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 >> 172.21.15.153:0/1396098512 conn(0x7fc9580fc7c0 msgr2=0x7fc9580fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:01:49.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 shutdown_connections 2024-03-16T11:01:49.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:01:49.835+0000 7fc960072700 1 -- 172.21.15.153:0/1396098512 wait complete. 2024-03-16T11:01:50.468 DEBUG:teuthology.orchestra.run.smithi153:osd.8> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.8.service 2024-03-16T11:01:50.470 INFO:tasks.cephadm:Deploying osd.9 on smithi078 with /dev/vg_nvme/lv_1... 2024-03-16T11:01:50.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T11:01:50.718 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:01:51.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:51.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:51.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:51.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:50 smithi153 ceph-mon[35521]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:51.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:50 smithi153 ceph-mon[35521]: from='osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:01:51.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:50 smithi153 ceph-mon[35521]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:01:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:50 smithi078 ceph-mon[32235]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:50 smithi078 ceph-mon[32235]: from='osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:01:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:50 smithi078 ceph-mon[32235]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:01:51.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:51.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:51.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:01:51.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:50 smithi098 ceph-mon[35848]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:51.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:50 smithi098 ceph-mon[35848]: from='osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:01:51.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:50 smithi098 ceph-mon[35848]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:01:52.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:51 smithi098 ceph-mon[35848]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T11:01:52.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:51 smithi098 ceph-mon[35848]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T11:01:52.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:52.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:51 smithi098 ceph-mon[35848]: from='osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:01:52.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:51 smithi098 ceph-mon[35848]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:01:52.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:51 smithi078 ceph-mon[32235]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T11:01:52.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:51 smithi078 ceph-mon[32235]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T11:01:52.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:52.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:51 smithi078 ceph-mon[32235]: from='osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:01:52.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:51 smithi078 ceph-mon[32235]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:01:52.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:51 smithi153 ceph-mon[35521]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T11:01:52.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:51 smithi153 ceph-mon[35521]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T11:01:52.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:52.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:51 smithi153 ceph-mon[35521]: from='osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:01:52.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:51 smithi153 ceph-mon[35521]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:01:52.294 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:01:51 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:01:51.832+0000 7f55d96d3700 -1 osd.8 0 waiting for initial osdmap 2024-03-16T11:01:52.294 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:01:51 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:01:51.844+0000 7f55d54c6700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:01:52.881 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T11:01:52.881 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T11:01:52.881 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-16T11:01:52.882 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-16T11:01:52.882 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114111 s, 919 MB/s 2024-03-16T11:01:52.882 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-16T11:01:53.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:01:53.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: osdmap e60: 9 total, 8 up, 9 in 2024-03-16T11:01:53.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] boot 2024-03-16T11:01:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T11:01:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:52 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:01:53.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: osdmap e60: 9 total, 8 up, 9 in 2024-03-16T11:01:53.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] boot 2024-03-16T11:01:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T11:01:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:52 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: osdmap e60: 9 total, 8 up, 9 in 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: osd.8 [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] boot 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T11:01:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:52 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:01:53.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2024-03-16T11:01:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:01:54.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:01:54.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:01:54.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:54.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:54.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:53 smithi078 ceph-mon[32235]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T11:01:54.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:01:54.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:01:54.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:53 smithi098 ceph-mon[35848]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T11:01:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:01:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:01:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:53 smithi153 ceph-mon[35521]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T11:01:54.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:54 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi153 2024-03-16T11:01:54.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:54 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi153 to 5696M 2024-03-16T11:01:54.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:54 smithi098 ceph-mon[35848]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:54.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:54 smithi098 ceph-mon[35848]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T11:01:55.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:54 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi153 2024-03-16T11:01:55.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:54 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi153 to 5696M 2024-03-16T11:01:55.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:54 smithi078 ceph-mon[32235]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:55.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:54 smithi078 ceph-mon[32235]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T11:01:55.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:54 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi153 2024-03-16T11:01:55.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:54 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi153 to 5696M 2024-03-16T11:01:55.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:54 smithi153 ceph-mon[35521]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:55.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:54 smithi153 ceph-mon[35521]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 -- 172.21.15.78:0/3822843697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c103ea0 msgr2=0x7f5d6c10a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/3822843697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c103ea0 0x7f5d6c10a750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400b0d0 tx=0x7f5d6400b3e0 comp rx=0 tx=0).stop 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 -- 172.21.15.78:0/3822843697 shutdown_connections 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/3822843697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d6c102300 0x7f5d6c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/3822843697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c103ea0 0x7f5d6c10a750 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/3822843697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d6c103500 0x7f5d6c103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 -- 172.21.15.78:0/3822843697 >> 172.21.15.78:0/3822843697 conn(0x7f5d6c0fd8e0 msgr2=0x7f5d6c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 -- 172.21.15.78:0/3822843697 shutdown_connections 2024-03-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 -- 172.21.15.78:0/3822843697 wait complete. 2024-03-16T11:01:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.569+0000 7f5d71c2c700 1 Processor -- start 2024-03-16T11:01:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d71c2c700 1 -- start start 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d71c2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c102300 0x7f5d6c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d71c2c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d6c103500 0x7f5d6c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d71c2c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d6c103ea0 0x7f5d6c19f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d71c2c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c10dd70 con 0x7f5d6c102300 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d71c2c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c10dbf0 con 0x7f5d6c103500 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d71c2c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c10def0 con 0x7f5d6c103ea0 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c102300 0x7f5d6c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c102300 0x7f5d6c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54998/0 (socket says 172.21.15.78:54998) 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 -- 172.21.15.78:0/2959543321 learned_addr learned my addr 172.21.15.78:0/2959543321 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6affd700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d6c103500 0x7f5d6c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6bfff700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d6c103ea0 0x7f5d6c19f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 -- 172.21.15.78:0/2959543321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d6c103ea0 msgr2=0x7f5d6c19f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d6c103ea0 0x7f5d6c19f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 -- 172.21.15.78:0/2959543321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d6c103500 msgr2=0x7f5d6c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d6c103500 0x7f5d6c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 -- 172.21.15.78:0/2959543321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6c19ffe0 con 0x7f5d6c102300 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.570+0000 7f5d6b7fe700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c102300 0x7f5d6c198380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00d9b0 tx=0x7f5d5c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.571+0000 7f5d68ff9700 1 -- 172.21.15.78:0/2959543321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c016070 con 0x7f5d6c102300 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.571+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d6c1a02d0 con 0x7f5d6c102300 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.571+0000 7f5d68ff9700 1 -- 172.21.15.78:0/2959543321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5d5c00be00 con 0x7f5d6c102300 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.571+0000 7f5d68ff9700 1 -- 172.21.15.78:0/2959543321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c010e30 con 0x7f5d6c102300 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.571+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d6c1a0860 con 0x7f5d6c102300 2024-03-16T11:01:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.572+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d6c068f90 con 0x7f5d6c102300 2024-03-16T11:01:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.573+0000 7f5d68ff9700 1 -- 172.21.15.78:0/2959543321 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5d5c019070 con 0x7f5d6c102300 2024-03-16T11:01:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.573+0000 7f5d68ff9700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5d540a2890 0x7f5d540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:01:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.573+0000 7f5d6affd700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5d540a2890 0x7f5d540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:01:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.573+0000 7f5d68ff9700 1 -- 172.21.15.78:0/2959543321 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5829+0+0 (secure 0 0 0) 0x7f5d5c0cdbf0 con 0x7f5d6c102300 2024-03-16T11:01:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.574+0000 7f5d6affd700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5d540a2890 0x7f5d540a4d50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005950 tx=0x7f5d6000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:01:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.578+0000 7f5d68ff9700 1 -- 172.21.15.78:0/2959543321 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d5c020030 con 0x7f5d6c102300 2024-03-16T11:01:55.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:01:55.784+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5d6c062710 con 0x7f5d540a2890 2024-03-16T11:01:56.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:56 smithi098 ceph-mon[35848]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T11:01:56.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:56 smithi098 ceph-mon[35848]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:56.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:56.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:56.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:56.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:56 smithi098 ceph-mon[35848]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:56.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:56 smithi078 ceph-mon[32235]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T11:01:56.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:56 smithi078 ceph-mon[32235]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:56.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:56.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:56.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:56.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:56 smithi078 ceph-mon[32235]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:56.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:56 smithi153 ceph-mon[35521]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T11:01:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:56 smithi153 ceph-mon[35521]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:01:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:01:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:01:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:01:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:56 smithi153 ceph-mon[35521]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:57.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:57 smithi098 ceph-mon[35848]: osdmap e65: 9 total, 9 up, 9 in 2024-03-16T11:01:57.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:57 smithi078 ceph-mon[32235]: osdmap e65: 9 total, 9 up, 9 in 2024-03-16T11:01:57.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:57 smithi153 ceph-mon[35521]: osdmap e65: 9 total, 9 up, 9 in 2024-03-16T11:01:58.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:58 smithi098 ceph-mon[35848]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:58.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:58 smithi153 ceph-mon[35521]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:58.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:58 smithi078 ceph-mon[32235]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:01:59.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1275589298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66171eea-9d12-44e4-b449-b83fa2a1d52a"}]: dispatch 2024-03-16T11:01:59.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1275589298' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66171eea-9d12-44e4-b449-b83fa2a1d52a"}]': finished 2024-03-16T11:01:59.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:59 smithi153 ceph-mon[35521]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T11:01:59.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:01:59.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:01:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2381223512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:59.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1275589298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66171eea-9d12-44e4-b449-b83fa2a1d52a"}]: dispatch 2024-03-16T11:01:59.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1275589298' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66171eea-9d12-44e4-b449-b83fa2a1d52a"}]': finished 2024-03-16T11:01:59.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:59 smithi078 ceph-mon[32235]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T11:01:59.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:01:59.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:01:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2381223512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:01:59.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1275589298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66171eea-9d12-44e4-b449-b83fa2a1d52a"}]: dispatch 2024-03-16T11:01:59.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1275589298' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66171eea-9d12-44e4-b449-b83fa2a1d52a"}]': finished 2024-03-16T11:01:59.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:59 smithi098 ceph-mon[35848]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T11:01:59.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:01:59.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:01:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2381223512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:02:00.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:00 smithi153 ceph-mon[35521]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-16T11:02:00.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:00 smithi078 ceph-mon[32235]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-16T11:02:00.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:00 smithi098 ceph-mon[35848]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-16T11:02:03.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:02 smithi078 ceph-mon[32235]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:02:03.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:02 smithi153 ceph-mon[35521]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:02:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:02 smithi098 ceph-mon[35848]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:02:05.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:04 smithi153 ceph-mon[35521]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:02:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:04 smithi078 ceph-mon[32235]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:02:05.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:04 smithi098 ceph-mon[35848]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:02:07.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:06 smithi078 ceph-mon[32235]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:02:07.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T11:02:07.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:06 smithi153 ceph-mon[35521]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:02:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T11:02:07.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:06 smithi098 ceph-mon[35848]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:02:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T11:02:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:08.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:08 smithi153 ceph-mon[35521]: Deploying daemon osd.9 on smithi078 2024-03-16T11:02:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:08 smithi078 ceph-mon[32235]: Deploying daemon osd.9 on smithi078 2024-03-16T11:02:08.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:08 smithi098 ceph-mon[35848]: Deploying daemon osd.9 on smithi078 2024-03-16T11:02:09.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:09 smithi153 ceph-mon[35521]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:02:09.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:09 smithi078 ceph-mon[32235]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:02:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:09 smithi098 ceph-mon[35848]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:02:11.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:11 smithi153 ceph-mon[35521]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:11.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:11 smithi078 ceph-mon[32235]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:11 smithi098 ceph-mon[35848]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:13.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:13 smithi078 ceph-mon[32235]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:13.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:13.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:13.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:13 smithi153 ceph-mon[35521]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:13 smithi098 ceph-mon[35848]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:15.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:15 smithi078 ceph-mon[32235]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:15.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:15 smithi153 ceph-mon[35521]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:15 smithi098 ceph-mon[35848]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:16.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.242+0000 7f5d68ff9700 1 -- 172.21.15.78:0/2959543321 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5d6c062710 con 0x7f5d540a2890 2024-03-16T11:02:16.244 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 9 on host 'smithi078' 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5d540a2890 msgr2=0x7f5d540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5d540a2890 0x7f5d540a4d50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005950 tx=0x7f5d6000a300 comp rx=0 tx=0).stop 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c102300 msgr2=0x7f5d6c198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c102300 0x7f5d6c198380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00d9b0 tx=0x7f5d5c00dcc0 comp rx=0 tx=0).stop 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 shutdown_connections 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5d540a2890 0x7f5d540a4d50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d6c103ea0 0x7f5d6c19f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d6c102300 0x7f5d6c198380 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 --2- 172.21.15.78:0/2959543321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d6c103500 0x7f5d6c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 >> 172.21.15.78:0/2959543321 conn(0x7f5d6c0fd8e0 msgr2=0x7f5d6c106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 shutdown_connections 2024-03-16T11:02:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:02:16.245+0000 7f5d71c2c700 1 -- 172.21.15.78:0/2959543321 wait complete. 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:16.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:16 smithi078 ceph-mon[32235]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T11:02:16.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:16.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:16 smithi098 ceph-mon[35848]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T11:02:16.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:16 smithi153 ceph-mon[35521]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T11:02:16.959 DEBUG:teuthology.orchestra.run.smithi078:osd.9> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.9.service 2024-03-16T11:02:16.962 INFO:tasks.cephadm:Deploying osd.10 on smithi098 with /dev/vg_nvme/lv_1... 2024-03-16T11:02:16.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T11:02:17.211 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:02:18.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:18 smithi098 ceph-mon[35848]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T11:02:18.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:18 smithi098 ceph-mon[35848]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T11:02:18.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:18 smithi098 ceph-mon[35848]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T11:02:18.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:18.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:18 smithi098 ceph-mon[35848]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:18 smithi153 ceph-mon[35521]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T11:02:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:18 smithi153 ceph-mon[35521]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T11:02:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:18 smithi153 ceph-mon[35521]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T11:02:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:18 smithi153 ceph-mon[35521]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:18 smithi078 ceph-mon[32235]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T11:02:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:18 smithi078 ceph-mon[32235]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T11:02:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:18 smithi078 ceph-mon[32235]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-16T11:02:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:18 smithi078 ceph-mon[32235]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T11:02:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T11:02:19.069 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T11:02:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T11:02:19.069 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112211 s, 934 MB/s 2024-03-16T11:02:19.069 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T11:02:19.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:19 smithi098 ceph-mon[35848]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T11:02:19.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:19 smithi098 ceph-mon[35848]: osdmap e68: 10 total, 9 up, 10 in 2024-03-16T11:02:19.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:19 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:19.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:19 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:19.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:19 smithi098 ceph-mon[35848]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' 2024-03-16T11:02:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:19 smithi153 ceph-mon[35521]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T11:02:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:19 smithi153 ceph-mon[35521]: osdmap e68: 10 total, 9 up, 10 in 2024-03-16T11:02:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:19 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:19.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:19 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:19.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:19 smithi153 ceph-mon[35521]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' 2024-03-16T11:02:19.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:19 smithi078 ceph-mon[32235]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-16T11:02:19.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:19 smithi078 ceph-mon[32235]: osdmap e68: 10 total, 9 up, 10 in 2024-03-16T11:02:19.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:19 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:19.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:19 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:19.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:19 smithi078 ceph-mon[32235]: from='osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708]' entity='osd.9' 2024-03-16T11:02:19.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:02:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:02:19.177+0000 7f15be046700 -1 osd.9 0 waiting for initial osdmap 2024-03-16T11:02:19.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:02:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:02:19.379+0000 7f15b5e31700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:02:19.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-03-16T11:02:19.890 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:02:20.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:20 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:02:20.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:20 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:02:20.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:20 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:20.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:20 smithi098 ceph-mon[35848]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:20 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:02:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:20 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:02:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:20 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:20 smithi153 ceph-mon[35521]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:20 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:02:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:20 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:02:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:20 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:20.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:20 smithi078 ceph-mon[32235]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:02:21.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 -- 172.21.15.98:0/1223145824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c102410 msgr2=0x7f822c102870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:21.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 --2- 172.21.15.98:0/1223145824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c102410 0x7f822c102870 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f82140099c0 tx=0x7f8214009cd0 comp rx=0 tx=0).stop 2024-03-16T11:02:21.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 -- 172.21.15.98:0/1223145824 shutdown_connections 2024-03-16T11:02:21.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 --2- 172.21.15.98:0/1223145824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f822c101210 0x7f822c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:21.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 --2- 172.21.15.98:0/1223145824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f822c102db0 0x7f822c109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:21.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 --2- 172.21.15.98:0/1223145824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c102410 0x7f822c102870 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:21.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 -- 172.21.15.98:0/1223145824 >> 172.21.15.98:0/1223145824 conn(0x7f822c0fc790 msgr2=0x7f822c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 -- 172.21.15.98:0/1223145824 shutdown_connections 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.422+0000 7f823262b700 1 -- 172.21.15.98:0/1223145824 wait complete. 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 Processor -- start 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 -- start start 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c101210 0x7f822c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f822c102410 0x7f822c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f822c102db0 0x7f822c19fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f822c10ccb0 con 0x7f822c102db0 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f822c10cb30 con 0x7f822c101210 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f823262b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f822c10ce30 con 0x7f822c102410 2024-03-16T11:02:21.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f822bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c101210 0x7f822c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f822bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c101210 0x7f822c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48306/0 (socket says 172.21.15.98:48306) 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f822bfff700 1 -- 172.21.15.98:0/1839437644 learned_addr learned my addr 172.21.15.98:0/1839437644 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.423+0000 7f8230bc8700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f822c102db0 0x7f822c19fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f822b7fe700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f822c102410 0x7f822c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f822bfff700 1 -- 172.21.15.98:0/1839437644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f822c102410 msgr2=0x7f822c1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f822bfff700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f822c102410 0x7f822c1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f822bfff700 1 -- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f822c102db0 msgr2=0x7f822c19fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f822bfff700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f822c102db0 0x7f822c19fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f822bfff700 1 -- 172.21.15.98:0/1839437644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822c1a00d0 con 0x7f822c101210 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f822bfff700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c101210 0x7f822c198470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f821c00d980 tx=0x7f821c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f82297fa700 1 -- 172.21.15.98:0/1839437644 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c017070 con 0x7f822c101210 2024-03-16T11:02:21.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f822c1a03c0 con 0x7f822c101210 2024-03-16T11:02:21.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f82297fa700 1 -- 172.21.15.98:0/1839437644 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f821c009de0 con 0x7f822c101210 2024-03-16T11:02:21.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f82297fa700 1 -- 172.21.15.98:0/1839437644 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c00f3c0 con 0x7f822c101210 2024-03-16T11:02:21.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.424+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f822c1a08b0 con 0x7f822c101210 2024-03-16T11:02:21.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.425+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f822c068f90 con 0x7f822c101210 2024-03-16T11:02:21.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.426+0000 7f82297fa700 1 -- 172.21.15.98:0/1839437644 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f821c009930 con 0x7f822c101210 2024-03-16T11:02:21.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.427+0000 7f82297fa700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82180a2890 0x7f82180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:21.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.427+0000 7f82297fa700 1 -- 172.21.15.98:0/1839437644 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7f821c0cdd50 con 0x7f822c101210 2024-03-16T11:02:21.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.427+0000 7f822b7fe700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82180a2890 0x7f82180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:21.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.427+0000 7f822b7fe700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82180a2890 0x7f82180a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82140038f0 tx=0x7f8214005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:02:21.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.430+0000 7f82297fa700 1 -- 172.21.15.98:0/1839437644 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f821c010460 con 0x7f822c101210 2024-03-16T11:02:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] boot 2024-03-16T11:02:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi078 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi078 to 4272M 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:21 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] boot 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi078 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi078 to 4272M 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:21.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:21 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: osd.9 [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] boot 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi078 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:21.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi078 to 4272M 2024-03-16T11:02:21.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:21.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:21.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:21 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:21.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:21.634+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f822c0fe9b0 con 0x7f82180a2890 2024-03-16T11:02:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:22 smithi153 ceph-mon[35521]: osdmap e70: 10 total, 10 up, 10 in 2024-03-16T11:02:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:22 smithi153 ceph-mon[35521]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:02:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:02:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:02:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:22 smithi153 ceph-mon[35521]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:22.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:22 smithi078 ceph-mon[32235]: osdmap e70: 10 total, 10 up, 10 in 2024-03-16T11:02:22.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:22 smithi078 ceph-mon[32235]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:02:22.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:02:22.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:02:22.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:22.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:22 smithi078 ceph-mon[32235]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:22 smithi098 ceph-mon[35848]: osdmap e70: 10 total, 10 up, 10 in 2024-03-16T11:02:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:22 smithi098 ceph-mon[35848]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:02:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:02:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:02:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:22 smithi098 ceph-mon[35848]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:25.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:25 smithi153 ceph-mon[35521]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:25.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:25 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/3303092175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]: dispatch 2024-03-16T11:02:25.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:25 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]: dispatch 2024-03-16T11:02:25.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:25 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]': finished 2024-03-16T11:02:25.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:25 smithi153 ceph-mon[35521]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T11:02:25.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:25.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:25 smithi153 ceph-mon[35521]: from='client.? 172.21.15.98:0/3453972528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:02:25.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:25 smithi078 ceph-mon[32235]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:25.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:25 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/3303092175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]: dispatch 2024-03-16T11:02:25.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:25 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]: dispatch 2024-03-16T11:02:25.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:25 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]': finished 2024-03-16T11:02:25.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:25 smithi078 ceph-mon[32235]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T11:02:25.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:25.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:25 smithi078 ceph-mon[32235]: from='client.? 172.21.15.98:0/3453972528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:02:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:25 smithi098 ceph-mon[35848]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:25.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:25 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/3303092175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]: dispatch 2024-03-16T11:02:25.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:25 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]: dispatch 2024-03-16T11:02:25.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:25 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72e85ff6-49af-4fc8-8114-9f33496e912b"}]': finished 2024-03-16T11:02:25.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:25 smithi098 ceph-mon[35848]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T11:02:25.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:25.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:25 smithi098 ceph-mon[35848]: from='client.? 172.21.15.98:0/3453972528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:02:27.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:27 smithi153 ceph-mon[35521]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:27.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:27 smithi078 ceph-mon[32235]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:27 smithi098 ceph-mon[35848]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:29.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:29 smithi098 ceph-mon[35848]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:29.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:29 smithi153 ceph-mon[35521]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:29.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:29 smithi078 ceph-mon[32235]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:31.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:31 smithi098 ceph-mon[35848]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:31.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:31 smithi153 ceph-mon[35521]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:31.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:31 smithi078 ceph-mon[32235]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T11:02:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T11:02:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:32.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T11:02:32.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:33.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:33 smithi098 ceph-mon[35848]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:33.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:33 smithi098 ceph-mon[35848]: Deploying daemon osd.10 on smithi098 2024-03-16T11:02:33.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:33 smithi153 ceph-mon[35521]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:33.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:33 smithi153 ceph-mon[35521]: Deploying daemon osd.10 on smithi098 2024-03-16T11:02:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:33 smithi078 ceph-mon[32235]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:33 smithi078 ceph-mon[32235]: Deploying daemon osd.10 on smithi098 2024-03-16T11:02:35.307 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:35 smithi098 ceph-mon[35848]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:35 smithi078 ceph-mon[32235]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:35 smithi153 ceph-mon[35521]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:37.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:37 smithi098 ceph-mon[35848]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:37.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:37 smithi078 ceph-mon[32235]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:37 smithi153 ceph-mon[35521]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:38.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:38.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:38.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:38 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:38.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:38 smithi098 ceph-mon[35848]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:38.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:38.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:38.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:38 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:38.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:38 smithi153 ceph-mon[35521]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:38.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:38.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:38.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:38 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:38.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:38 smithi078 ceph-mon[32235]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:40.800 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 10 on host 'smithi098' 2024-03-16T11:02:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.799+0000 7f82297fa700 1 -- 172.21.15.98:0/1839437644 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f822c0fe9b0 con 0x7f82180a2890 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82180a2890 msgr2=0x7f82180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82180a2890 0x7f82180a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82140038f0 tx=0x7f8214005fd0 comp rx=0 tx=0).stop 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c101210 msgr2=0x7f822c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c101210 0x7f822c198470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f821c00d980 tx=0x7f821c00dc90 comp rx=0 tx=0).stop 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 shutdown_connections 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82180a2890 0x7f82180a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f822c102db0 0x7f822c19fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f822c101210 0x7f822c198470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:40.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 --2- 172.21.15.98:0/1839437644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f822c102410 0x7f822c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:40.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.801+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 >> 172.21.15.98:0/1839437644 conn(0x7f822c0fc790 msgr2=0x7f822c0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:02:40.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.802+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 shutdown_connections 2024-03-16T11:02:40.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T11:02:40.802+0000 7f823262b700 1 -- 172.21.15.98:0/1839437644 wait complete. 2024-03-16T11:02:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:41 smithi153 ceph-mon[35521]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:41 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:41 smithi078 ceph-mon[32235]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:41 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:41.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:41 smithi098 ceph-mon[35848]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:41.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:41.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:41 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:41.470 DEBUG:teuthology.orchestra.run.smithi098:osd.10> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.10.service 2024-03-16T11:02:41.475 INFO:tasks.cephadm:Deploying osd.11 on smithi153 with /dev/vg_nvme/lv_1... 2024-03-16T11:02:41.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T11:02:41.720 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:02:42.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:42 smithi153 ceph-mon[35521]: from='osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:02:42.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:42 smithi153 ceph-mon[35521]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:02:42.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:42.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:42.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:42 smithi098 ceph-mon[35848]: from='osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:02:42.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:42 smithi098 ceph-mon[35848]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:02:42.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:42.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:42.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:42 smithi078 ceph-mon[32235]: from='osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:02:42.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:42 smithi078 ceph-mon[32235]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:02:42.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:42.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:43.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:43 smithi153 ceph-mon[35521]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:43.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:43 smithi153 ceph-mon[35521]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T11:02:43.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:43 smithi153 ceph-mon[35521]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T11:02:43.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:43 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:43.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:43 smithi153 ceph-mon[35521]: from='osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:02:43.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:43 smithi153 ceph-mon[35521]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:02:43.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:43 smithi078 ceph-mon[32235]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:43.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:43 smithi078 ceph-mon[32235]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T11:02:43.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:43 smithi078 ceph-mon[32235]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T11:02:43.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:43 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:43.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:43 smithi078 ceph-mon[32235]: from='osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:02:43.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:43 smithi078 ceph-mon[32235]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:02:43.314 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:43 smithi098 ceph-mon[35848]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:43.314 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:43 smithi098 ceph-mon[35848]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T11:02:43.314 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:43 smithi098 ceph-mon[35848]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T11:02:43.314 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:43 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:43.314 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:43 smithi098 ceph-mon[35848]: from='osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:02:43.314 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:43 smithi098 ceph-mon[35848]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T11:02:43.804 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T11:02:43.804 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T11:02:43.804 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T11:02:43.804 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T11:02:43.804 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123253 s, 851 MB/s 2024-03-16T11:02:43.804 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 2024-03-16T11:02:43.804 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T11:02:44.190 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:44 smithi153 ceph-mon[35521]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:02:44.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:44 smithi153 ceph-mon[35521]: osdmap e73: 11 total, 10 up, 11 in 2024-03-16T11:02:44.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:44 smithi153 ceph-mon[35521]: osdmap e74: 11 total, 10 up, 11 in 2024-03-16T11:02:44.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:44 smithi153 ceph-mon[35521]: from='osd.10 ' entity='osd.10' 2024-03-16T11:02:44.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-mon[35848]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:02:44.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-mon[35848]: osdmap e73: 11 total, 10 up, 11 in 2024-03-16T11:02:44.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-mon[35848]: osdmap e74: 11 total, 10 up, 11 in 2024-03-16T11:02:44.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-mon[35848]: from='osd.10 ' entity='osd.10' 2024-03-16T11:02:44.215 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:02:43 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:02:43.933+0000 7f9af7216700 -1 osd.10 0 waiting for initial osdmap 2024-03-16T11:02:44.215 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:02:44 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:02:44.076+0000 7f9aef001700 -1 osd.10 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:02:44.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:44 smithi078 ceph-mon[32235]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T11:02:44.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:44 smithi078 ceph-mon[32235]: osdmap e73: 11 total, 10 up, 11 in 2024-03-16T11:02:44.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:44 smithi078 ceph-mon[32235]: osdmap e74: 11 total, 10 up, 11 in 2024-03-16T11:02:44.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:44.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:44 smithi078 ceph-mon[32235]: from='osd.10 ' entity='osd.10' 2024-03-16T11:02:44.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-03-16T11:02:44.835 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:02:45.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:45 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:02:45.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:45 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:02:45.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:45 smithi153 ceph-mon[35521]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:45.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:45 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:45.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:45 smithi153 ceph-mon[35521]: osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] boot 2024-03-16T11:02:45.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:45 smithi153 ceph-mon[35521]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T11:02:45.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:45 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:45 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:02:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:45 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:02:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:45 smithi078 ceph-mon[32235]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:45 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:45 smithi078 ceph-mon[32235]: osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] boot 2024-03-16T11:02:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:45 smithi078 ceph-mon[32235]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T11:02:45.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:45 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:45 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:02:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:45 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:02:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:45 smithi098 ceph-mon[35848]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:02:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:45 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:45 smithi098 ceph-mon[35848]: osd.10 [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] boot 2024-03-16T11:02:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:45 smithi098 ceph-mon[35848]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T11:02:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:45 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:02:46.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi098 2024-03-16T11:02:46.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi098 to 4272M 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:46.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:46.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:46.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:46.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:46.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:46 smithi153 ceph-mon[35521]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T11:02:46.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi098 2024-03-16T11:02:46.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi098 to 4272M 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:46.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:46.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:46.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:46.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:46.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:46 smithi078 ceph-mon[32235]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T11:02:46.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi098 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi098 to 4272M 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:02:46.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:46.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:02:46.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:02:46.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:46 smithi098 ceph-mon[35848]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T11:02:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 -- 172.21.15.153:0/1374357153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501035d0 msgr2=0x7efe50103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 --2- 172.21.15.153:0/1374357153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501035d0 0x7efe50103a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe38009990 tx=0x7efe38009ca0 comp rx=0 tx=0).stop 2024-03-16T11:02:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 -- 172.21.15.153:0/1374357153 shutdown_connections 2024-03-16T11:02:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 --2- 172.21.15.153:0/1374357153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe50103f70 0x7efe5010a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 --2- 172.21.15.153:0/1374357153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe501005f0 0x7efe50100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 --2- 172.21.15.153:0/1374357153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501035d0 0x7efe50103a30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 -- 172.21.15.153:0/1374357153 >> 172.21.15.153:0/1374357153 conn(0x7efe500fbf70 msgr2=0x7efe500fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:02:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 -- 172.21.15.153:0/1374357153 shutdown_connections 2024-03-16T11:02:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 -- 172.21.15.153:0/1374357153 wait complete. 2024-03-16T11:02:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.498+0000 7efe553b3700 1 Processor -- start 2024-03-16T11:02:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 -- start start 2024-03-16T11:02:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501005f0 0x7efe50198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe501035d0 0x7efe50198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe50103f70 0x7efe5019fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efe5010deb0 con 0x7efe501035d0 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe5010dd30 con 0x7efe50103f70 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe5010e030 con 0x7efe501005f0 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501005f0 0x7efe50198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501005f0 0x7efe50198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44604/0 (socket says 172.21.15.153:44604) 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 -- 172.21.15.153:0/3394622739 learned_addr learned my addr 172.21.15.153:0/3394622739 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4f7fe700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe50103f70 0x7efe5019fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe467fc700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe501035d0 0x7efe50198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 -- 172.21.15.153:0/3394622739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe50103f70 msgr2=0x7efe5019fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe50103f70 0x7efe5019fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 -- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe501035d0 msgr2=0x7efe50198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe501035d0 0x7efe50198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 -- 172.21.15.153:0/3394622739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe501a0180 con 0x7efe501005f0 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4f7fe700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe50103f70 0x7efe5019fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4effd700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501005f0 0x7efe50198520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe4000b530 tx=0x7efe4000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4cff9700 1 -- 172.21.15.153:0/3394622739 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe40015020 con 0x7efe501005f0 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe501a0470 con 0x7efe501005f0 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.499+0000 7efe4cff9700 1 -- 172.21.15.153:0/3394622739 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe40011ce0 con 0x7efe501005f0 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.500+0000 7efe4cff9700 1 -- 172.21.15.153:0/3394622739 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe40010d60 con 0x7efe501005f0 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.500+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe501a0a30 con 0x7efe501005f0 2024-03-16T11:02:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.501+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe50068f90 con 0x7efe501005f0 2024-03-16T11:02:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.502+0000 7efe4cff9700 1 -- 172.21.15.153:0/3394622739 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efe40018020 con 0x7efe501005f0 2024-03-16T11:02:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.502+0000 7efe4cff9700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe3c0a2890 0x7efe3c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:02:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.502+0000 7efe4cff9700 1 -- 172.21.15.153:0/3394622739 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6723+0+0 (secure 0 0 0) 0x7efe400cdc30 con 0x7efe501005f0 2024-03-16T11:02:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.502+0000 7efe467fc700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe3c0a2890 0x7efe3c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:02:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.503+0000 7efe467fc700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe3c0a2890 0x7efe3c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe38006010 tx=0x7efe38005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:02:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.507+0000 7efe4cff9700 1 -- 172.21.15.153:0/3394622739 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe4001f020 con 0x7efe501005f0 2024-03-16T11:02:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:02:46.711+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7efe50062710 con 0x7efe3c0a2890 2024-03-16T11:02:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:47 smithi153 ceph-mon[35521]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:47 smithi153 ceph-mon[35521]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T11:02:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:02:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:02:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:47.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:47 smithi078 ceph-mon[32235]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:47.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:47 smithi078 ceph-mon[32235]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T11:02:47.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:02:47.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:02:47.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:47 smithi098 ceph-mon[35848]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:47 smithi098 ceph-mon[35848]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T11:02:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:02:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:02:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:48 smithi153 ceph-mon[35521]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:02:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:48 smithi153 ceph-mon[35521]: osdmap e78: 11 total, 11 up, 11 in 2024-03-16T11:02:48.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:48 smithi078 ceph-mon[32235]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:02:48.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:48 smithi078 ceph-mon[32235]: osdmap e78: 11 total, 11 up, 11 in 2024-03-16T11:02:48.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:48 smithi098 ceph-mon[35848]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:02:48.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:48 smithi098 ceph-mon[35848]: osdmap e78: 11 total, 11 up, 11 in 2024-03-16T11:02:49.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:49 smithi153 ceph-mon[35521]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:49.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:49 smithi153 ceph-mon[35521]: osdmap e79: 11 total, 11 up, 11 in 2024-03-16T11:02:49.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:49 smithi078 ceph-mon[32235]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:49.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:49 smithi078 ceph-mon[32235]: osdmap e79: 11 total, 11 up, 11 in 2024-03-16T11:02:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:49 smithi098 ceph-mon[35848]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:49.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:49 smithi098 ceph-mon[35848]: osdmap e79: 11 total, 11 up, 11 in 2024-03-16T11:02:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:50 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/2401833501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]: dispatch 2024-03-16T11:02:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:50 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]: dispatch 2024-03-16T11:02:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:50 smithi078 ceph-mon[32235]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]': finished 2024-03-16T11:02:50.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:50 smithi078 ceph-mon[32235]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T11:02:50.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:02:50.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:50 smithi078 ceph-mon[32235]: from='client.? 172.21.15.153:0/1701203347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:02:50.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:50 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/2401833501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]: dispatch 2024-03-16T11:02:50.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:50 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]: dispatch 2024-03-16T11:02:50.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:50 smithi098 ceph-mon[35848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]': finished 2024-03-16T11:02:50.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:50 smithi098 ceph-mon[35848]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T11:02:50.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:02:50.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:50 smithi098 ceph-mon[35848]: from='client.? 172.21.15.153:0/1701203347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:02:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:50 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/2401833501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]: dispatch 2024-03-16T11:02:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:50 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]: dispatch 2024-03-16T11:02:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:50 smithi153 ceph-mon[35521]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e383aa05-dedb-4d76-9e7a-0411d04ef5ab"}]': finished 2024-03-16T11:02:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:50 smithi153 ceph-mon[35521]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T11:02:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:02:50.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:50 smithi153 ceph-mon[35521]: from='client.? 172.21.15.153:0/1701203347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:02:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:51 smithi098 ceph-mon[35848]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T11:02:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:51 smithi153 ceph-mon[35521]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T11:02:51.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:51 smithi078 ceph-mon[32235]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T11:02:53.160 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:53 smithi153 ceph-mon[35521]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-16T11:02:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:53 smithi098 ceph-mon[35848]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-16T11:02:53.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:53 smithi078 ceph-mon[32235]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-16T11:02:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:55 smithi098 ceph-mon[35848]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:02:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:55 smithi153 ceph-mon[35521]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:02:55.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:55 smithi078 ceph-mon[32235]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T11:02:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:57 smithi098 ceph-mon[35848]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:02:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T11:02:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:57.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:57 smithi153 ceph-mon[35521]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:02:57.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T11:02:57.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:57.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:57 smithi078 ceph-mon[32235]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:02:57.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T11:02:57.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:02:58.222 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:58 smithi153 ceph-mon[35521]: Deploying daemon osd.11 on smithi153 2024-03-16T11:02:58.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:58 smithi098 ceph-mon[35848]: Deploying daemon osd.11 on smithi153 2024-03-16T11:02:58.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:58 smithi078 ceph-mon[32235]: Deploying daemon osd.11 on smithi153 2024-03-16T11:02:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:02:59 smithi098 ceph-mon[35848]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:02:59 smithi153 ceph-mon[35521]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:02:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:02:59 smithi078 ceph-mon[32235]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:01.212 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:01 smithi153 ceph-mon[35521]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:01 smithi098 ceph-mon[35848]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:01 smithi078 ceph-mon[32235]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:03.352 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:03 smithi153 ceph-mon[35521]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:03.352 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:03.352 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:03.352 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:03 smithi098 ceph-mon[35848]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:03 smithi078 ceph-mon[32235]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:05.196 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:05 smithi153 ceph-mon[35521]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:05.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:05 smithi098 ceph-mon[35848]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:05.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:05 smithi078 ceph-mon[32235]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:06.127 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 11 on host 'smithi153' 2024-03-16T11:03:06.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.127+0000 7efe4cff9700 1 -- 172.21.15.153:0/3394622739 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7efe50062710 con 0x7efe3c0a2890 2024-03-16T11:03:06.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe3c0a2890 msgr2=0x7efe3c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:06.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe3c0a2890 0x7efe3c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe38006010 tx=0x7efe38005ab0 comp rx=0 tx=0).stop 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501005f0 msgr2=0x7efe50198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501005f0 0x7efe50198520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe4000b530 tx=0x7efe4000b8f0 comp rx=0 tx=0).stop 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 shutdown_connections 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe3c0a2890 0x7efe3c0a4d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501005f0 0x7efe50198520 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe50103f70 0x7efe5019fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 --2- 172.21.15.153:0/3394622739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe501035d0 0x7efe50198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 >> 172.21.15.153:0/3394622739 conn(0x7efe500fbf70 msgr2=0x7efe501068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 shutdown_connections 2024-03-16T11:03:06.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:06.129+0000 7efe553b3700 1 -- 172.21.15.153:0/3394622739 wait complete. 2024-03-16T11:03:06.486 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.486 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:03:06.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:06 smithi153 ceph-mon[35521]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:03:06.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:06.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:06.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:06.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:06.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:03:06.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:06 smithi098 ceph-mon[35848]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:06.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:06.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:03:06.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:06 smithi078 ceph-mon[32235]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:03:06.899 DEBUG:teuthology.orchestra.run.smithi153:osd.11> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.11.service 2024-03-16T11:03:06.901 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-16T11:03:06.901 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T11:03:07.157 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:03:08.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:08 smithi078 ceph-mon[32235]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T11:03:08.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:08 smithi078 ceph-mon[32235]: osdmap e81: 12 total, 11 up, 12 in 2024-03-16T11:03:08.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:08 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:08.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:08 smithi078 ceph-mon[32235]: from='osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:03:08.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:08 smithi078 ceph-mon[32235]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:03:08.373 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:08 smithi153 ceph-mon[35521]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T11:03:08.373 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:08 smithi153 ceph-mon[35521]: osdmap e81: 12 total, 11 up, 12 in 2024-03-16T11:03:08.373 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:08 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:08.373 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:08 smithi153 ceph-mon[35521]: from='osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:03:08.373 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:08 smithi153 ceph-mon[35521]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:03:08.374 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:03:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:03:08.112+0000 7f5b85700700 -1 osd.11 0 waiting for initial osdmap 2024-03-16T11:03:08.374 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:03:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:03:08.133+0000 7f5b814f3700 -1 osd.11 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:03:08.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:08 smithi098 ceph-mon[35848]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T11:03:08.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:08 smithi098 ceph-mon[35848]: osdmap e81: 12 total, 11 up, 12 in 2024-03-16T11:03:08.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:08 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:08.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:08 smithi098 ceph-mon[35848]: from='osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:03:08.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:08 smithi098 ceph-mon[35848]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T11:03:08.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 -- 172.21.15.78:0/1989950075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 msgr2=0x7fc358102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/1989950075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc358102860 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0099c0 tx=0x7fc34c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 -- 172.21.15.78:0/1989950075 shutdown_connections 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/1989950075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc358101200 0x7fc358101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/1989950075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc358102da0 0x7fc358109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/1989950075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc358102860 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 -- 172.21.15.78:0/1989950075 >> 172.21.15.78:0/1989950075 conn(0x7fc3580fc7c0 msgr2=0x7fc3580febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 -- 172.21.15.78:0/1989950075 shutdown_connections 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.824+0000 7fc35c9dd700 1 -- 172.21.15.78:0/1989950075 wait complete. 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 Processor -- start 2024-03-16T11:03:08.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 -- start start 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc358101200 0x7fc358197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc3581978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc358102da0 0x7fc35819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc35810ccf0 con 0x7fc358102400 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc35810cb70 con 0x7fc358101200 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc35c9dd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc35810ce70 con 0x7fc358102da0 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.825+0000 7fc355d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc3581978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:08.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc3581978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36262/0 (socket says 172.21.15.78:36262) 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 -- 172.21.15.78:0/2755582905 learned_addr learned my addr 172.21.15.78:0/2755582905 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc356d9d700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc358102da0 0x7fc35819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc35659c700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc358101200 0x7fc358197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 -- 172.21.15.78:0/2755582905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc358102da0 msgr2=0x7fc35819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc358102da0 0x7fc35819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 -- 172.21.15.78:0/2755582905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc358101200 msgr2=0x7fc358197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc358101200 0x7fc358197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 -- 172.21.15.78:0/2755582905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35819efc0 con 0x7fc358102400 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc355d9b700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc3581978a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00fa30 tx=0x7fc34c00fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc3477fe700 1 -- 172.21.15.78:0/2755582905 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c027070 con 0x7fc358102400 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35819f250 con 0x7fc358102400 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc3477fe700 1 -- 172.21.15.78:0/2755582905 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc34c022920 con 0x7fc358102400 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc3477fe700 1 -- 172.21.15.78:0/2755582905 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c009e60 con 0x7fc358102400 2024-03-16T11:03:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.826+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35819f6b0 con 0x7fc358102400 2024-03-16T11:03:08.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.827+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc358068f90 con 0x7fc358102400 2024-03-16T11:03:08.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.828+0000 7fc3477fe700 1 -- 172.21.15.78:0/2755582905 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc34c02a070 con 0x7fc358102400 2024-03-16T11:03:08.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.829+0000 7fc3477fe700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc3400a2910 0x7fc3400a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:08.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.829+0000 7fc35659c700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc3400a2910 0x7fc3400a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:08.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.829+0000 7fc3477fe700 1 -- 172.21.15.78:0/2755582905 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6847+0+0 (secure 0 0 0) 0x7fc34c0cf4b0 con 0x7fc358102400 2024-03-16T11:03:08.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.830+0000 7fc35659c700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc3400a2910 0x7fc3400a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc358102260 tx=0x7fc348009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:08.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:08.833+0000 7fc3477fe700 1 -- 172.21.15.78:0/2755582905 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc34c013070 con 0x7fc358102400 2024-03-16T11:03:09.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.028+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fc35802db70 con 0x7fc358102400 2024-03-16T11:03:09.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.029+0000 7fc3477fe700 1 -- 172.21.15.78:0/2755582905 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7fc34c09cb00 con 0x7fc358102400 2024-03-16T11:03:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:09.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.031+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc3400a2910 msgr2=0x7fc3400a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:09.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.031+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc3400a2910 0x7fc3400a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc358102260 tx=0x7fc348009260 comp rx=0 tx=0).stop 2024-03-16T11:03:09.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.031+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 msgr2=0x7fc3581978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:09.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.031+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc3581978a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00fa30 tx=0x7fc34c00fa60 comp rx=0 tx=0).stop 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 shutdown_connections 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc3400a2910 0x7fc3400a4dd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc358102da0 0x7fc35819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc358101200 0x7fc358197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 --2- 172.21.15.78:0/2755582905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc358102400 0x7fc3581978a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 >> 172.21.15.78:0/2755582905 conn(0x7fc3580fc7c0 msgr2=0x7fc358105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 shutdown_connections 2024-03-16T11:03:09.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:09.032+0000 7fc35c9dd700 1 -- 172.21.15.78:0/2755582905 wait complete. 2024-03-16T11:03:09.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:09 smithi078 ceph-mon[32235]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:09.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:09 smithi078 ceph-mon[32235]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:03:09.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:09 smithi078 ceph-mon[32235]: osdmap e82: 12 total, 11 up, 12 in 2024-03-16T11:03:09.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:09 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:09.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:09 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:09.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:09 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2755582905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:03:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:09 smithi098 ceph-mon[35848]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:09.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:09 smithi098 ceph-mon[35848]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:03:09.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:09 smithi098 ceph-mon[35848]: osdmap e82: 12 total, 11 up, 12 in 2024-03-16T11:03:09.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:09 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:09.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:09 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:09.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:09 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2755582905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:03:09.499 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:09 smithi153 ceph-mon[35521]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:03:09.500 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:09 smithi153 ceph-mon[35521]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T11:03:09.500 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:09 smithi153 ceph-mon[35521]: osdmap e82: 12 total, 11 up, 12 in 2024-03-16T11:03:09.500 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:09 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:09.500 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:09 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:09.500 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:09 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2755582905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:03:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":82,"num_osds":12,"num_up_osds":11,"osd_up_since":1710586964,"num_in_osds":12,"osd_in_since":1710586969,"num_remapped_pgs":0} 2024-03-16T11:03:10.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: purged_snaps scrub starts 2024-03-16T11:03:10.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: purged_snaps scrub ok 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] boot 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:10 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: purged_snaps scrub starts 2024-03-16T11:03:10.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: purged_snaps scrub ok 2024-03-16T11:03:10.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] boot 2024-03-16T11:03:10.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T11:03:10.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:10.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:10 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: purged_snaps scrub starts 2024-03-16T11:03:10.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: purged_snaps scrub ok 2024-03-16T11:03:10.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: osd.11 [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] boot 2024-03-16T11:03:10.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T11:03:10.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:10.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:10 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:10.730 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T11:03:10.972 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:03:11.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:11 smithi078 ceph-mon[32235]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:11.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:11 smithi078 ceph-mon[32235]: Detected new or changed devices on smithi153 2024-03-16T11:03:11.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:11 smithi078 ceph-mon[32235]: Adjusting osd_memory_target on smithi153 to 4272M 2024-03-16T11:03:11.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:11 smithi078 ceph-mon[32235]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T11:03:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:11 smithi098 ceph-mon[35848]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:11 smithi098 ceph-mon[35848]: Detected new or changed devices on smithi153 2024-03-16T11:03:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:11 smithi098 ceph-mon[35848]: Adjusting osd_memory_target on smithi153 to 4272M 2024-03-16T11:03:11.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:11 smithi098 ceph-mon[35848]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T11:03:11.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:11 smithi153 ceph-mon[35521]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:11.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:11 smithi153 ceph-mon[35521]: Detected new or changed devices on smithi153 2024-03-16T11:03:11.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:11 smithi153 ceph-mon[35521]: Adjusting osd_memory_target on smithi153 to 4272M 2024-03-16T11:03:11.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:11 smithi153 ceph-mon[35521]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 -- 172.21.15.78:0/3654687940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c101200 msgr2=0x7fe48c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 --2- 172.21.15.78:0/3654687940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c101200 0x7fe48c101620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0099c0 tx=0x7fe47c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 -- 172.21.15.78:0/3654687940 shutdown_connections 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 --2- 172.21.15.78:0/3654687940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe48c102da0 0x7fe48c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 --2- 172.21.15.78:0/3654687940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c101200 0x7fe48c101620 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 --2- 172.21.15.78:0/3654687940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe48c102400 0x7fe48c102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 -- 172.21.15.78:0/3654687940 >> 172.21.15.78:0/3654687940 conn(0x7fe48c0fc7a0 msgr2=0x7fe48c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 -- 172.21.15.78:0/3654687940 shutdown_connections 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.653+0000 7fe49403c700 1 -- 172.21.15.78:0/3654687940 wait complete. 2024-03-16T11:03:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 Processor -- start 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 -- start start 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe48c101200 0x7fe48c06aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c102400 0x7fe48c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe48c102da0 0x7fe48c1a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c10cca0 con 0x7fe48c102400 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c10cb20 con 0x7fe48c102da0 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe49403c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c10ce20 con 0x7fe48c101200 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c102400 0x7fe48c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c102400 0x7fe48c19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36276/0 (socket says 172.21.15.78:36276) 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 -- 172.21.15.78:0/866683550 learned_addr learned my addr 172.21.15.78:0/866683550 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe491dd8700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe48c101200 0x7fe48c06aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4925d9700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe48c102da0 0x7fe48c1a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 -- 172.21.15.78:0/866683550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe48c101200 msgr2=0x7fe48c06aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe48c101200 0x7fe48c06aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 -- 172.21.15.78:0/866683550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe48c102da0 msgr2=0x7fe48c1a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe48c102da0 0x7fe48c1a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.654+0000 7fe4915d7700 1 -- 172.21.15.78:0/866683550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48c1a2340 con 0x7fe48c102400 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.655+0000 7fe4915d7700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c102400 0x7fe48c19f610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c920 tx=0x7fe48800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.655+0000 7fe482ffd700 1 -- 172.21.15.78:0/866683550 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488007b00 con 0x7fe48c102400 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.655+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe48c1a2630 con 0x7fe48c102400 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.655+0000 7fe482ffd700 1 -- 172.21.15.78:0/866683550 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe488004ce0 con 0x7fe48c102400 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.655+0000 7fe482ffd700 1 -- 172.21.15.78:0/866683550 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488005550 con 0x7fe48c102400 2024-03-16T11:03:12.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.655+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe48c1a2bc0 con 0x7fe48c102400 2024-03-16T11:03:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.656+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe48c068f90 con 0x7fe48c102400 2024-03-16T11:03:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.658+0000 7fe482ffd700 1 -- 172.21.15.78:0/866683550 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe488004750 con 0x7fe48c102400 2024-03-16T11:03:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.658+0000 7fe482ffd700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe4780a2910 0x7fe4780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.658+0000 7fe491dd8700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe4780a2910 0x7fe4780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.658+0000 7fe482ffd700 1 -- 172.21.15.78:0/866683550 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fe4880ce470 con 0x7fe48c102400 2024-03-16T11:03:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.659+0000 7fe491dd8700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe4780a2910 0x7fe4780a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe47c000c00 tx=0x7fe47c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:12.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.661+0000 7fe482ffd700 1 -- 172.21.15.78:0/866683550 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe488014030 con 0x7fe48c102400 2024-03-16T11:03:12.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.855+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fe48c062ee0 con 0x7fe48c102400 2024-03-16T11:03:12.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.855+0000 7fe482ffd700 1 -- 172.21.15.78:0/866683550 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v84) ==== 74+0+133 (secure 0 0 0) 0x7fe48809ba30 con 0x7fe48c102400 2024-03-16T11:03:12.856 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.857+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe4780a2910 msgr2=0x7fe4780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.857+0000 7fe49403c700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe4780a2910 0x7fe4780a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe47c000c00 tx=0x7fe47c005fb0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.857+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c102400 msgr2=0x7fe48c19f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c102400 0x7fe48c19f610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c920 tx=0x7fe48800cc30 comp rx=0 tx=0).stop 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 shutdown_connections 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe4780a2910 0x7fe4780a4dd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe48c101200 0x7fe48c06aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe48c102da0 0x7fe48c1a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 --2- 172.21.15.78:0/866683550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe48c102400 0x7fe48c19f610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 >> 172.21.15.78:0/866683550 conn(0x7fe48c0fc7a0 msgr2=0x7fe48c105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 shutdown_connections 2024-03-16T11:03:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:12.858+0000 7fe49403c700 1 -- 172.21.15.78:0/866683550 wait complete. 2024-03-16T11:03:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:13 smithi098 ceph-mon[35848]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:13 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/866683550' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:03:13.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:13 smithi153 ceph-mon[35521]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:13.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:13 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/866683550' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:03:13.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:13 smithi078 ceph-mon[32235]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:13.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:13 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/866683550' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:03:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"num_osds":12,"num_up_osds":12,"osd_up_since":1710586989,"num_in_osds":12,"osd_in_since":1710586969,"num_remapped_pgs":0} 2024-03-16T11:03:13.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T11:03:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:03:15.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:15 smithi078 ceph-mon[32235]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:15 smithi098 ceph-mon[35848]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1258723039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc074101e50 msgr2=0x7fc074105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1258723039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc074101e50 0x7fc074105ea0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc0700099c0 tx=0x7fc070009cd0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1258723039 shutdown_connections 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1258723039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc074106580 0x7fc07410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1258723039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0741014a0 0x7fc074101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1258723039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc074101e50 0x7fc074105ea0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1258723039 >> 172.21.15.78:0/1258723039 conn(0x7fc074078300 msgr2=0x7fc074078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1258723039 shutdown_connections 2024-03-16T11:03:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.520+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1258723039 wait complete. 2024-03-16T11:03:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 Processor -- start 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 -- start start 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0741014a0 0x7fc07419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc074101e50 0x7fc07419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc074106580 0x7fc074197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc074111410 con 0x7fc0741014a0 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc074111290 con 0x7fc074101e50 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc07b5e7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc074111590 con 0x7fc074106580 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc079383700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0741014a0 0x7fc07419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc079383700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0741014a0 0x7fc07419d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60108/0 (socket says 172.21.15.78:60108) 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc079383700 1 -- 172.21.15.78:0/1990063933 learned_addr learned my addr 172.21.15.78:0/1990063933 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.521+0000 7fc079b84700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc074106580 0x7fc074197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc078b82700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc074101e50 0x7fc07419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc079383700 1 -- 172.21.15.78:0/1990063933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc074106580 msgr2=0x7fc074197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc079383700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc074106580 0x7fc074197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc079383700 1 -- 172.21.15.78:0/1990063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc074101e50 msgr2=0x7fc07419db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc079383700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc074101e50 0x7fc07419db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc079383700 1 -- 172.21.15.78:0/1990063933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc074198050 con 0x7fc0741014a0 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc078b82700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc074101e50 0x7fc07419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc079383700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0741014a0 0x7fc07419d650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc064009970 tx=0x7fc064009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc06a7fc700 1 -- 172.21.15.78:0/1990063933 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06401c070 con 0x7fc0741014a0 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc06a7fc700 1 -- 172.21.15.78:0/1990063933 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc06400d260 con 0x7fc0741014a0 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0741982e0 con 0x7fc0741014a0 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc06a7fc700 1 -- 172.21.15.78:0/1990063933 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064017a00 con 0x7fc0741014a0 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.522+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc07406abe0 con 0x7fc0741014a0 2024-03-16T11:03:15.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.523+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc074102980 con 0x7fc0741014a0 2024-03-16T11:03:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.524+0000 7fc06a7fc700 1 -- 172.21.15.78:0/1990063933 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc06401f070 con 0x7fc0741014a0 2024-03-16T11:03:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.525+0000 7fc06a7fc700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc0600a2910 0x7fc0600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.525+0000 7fc078b82700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc0600a2910 0x7fc0600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.525+0000 7fc06a7fc700 1 -- 172.21.15.78:0/1990063933 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fc0640cf680 con 0x7fc0741014a0 2024-03-16T11:03:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.525+0000 7fc078b82700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc0600a2910 0x7fc0600a4dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc070000c00 tx=0x7fc070005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:15.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.528+0000 7fc06a7fc700 1 -- 172.21.15.78:0/1990063933 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc06409cbc0 con 0x7fc0741014a0 2024-03-16T11:03:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:15 smithi153 ceph-mon[35521]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:15.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.723+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc074198c50 con 0x7fc0741014a0 2024-03-16T11:03:15.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.724+0000 7fc06a7fc700 1 -- 172.21.15.78:0/1990063933 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18373 (secure 0 0 0) 0x7fc06409c4c0 con 0x7fc0741014a0 2024-03-16T11:03:15.725 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:15.725 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:03:10.113262+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:03:15.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc0600a2910 msgr2=0x7fc0600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:15.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc0600a2910 0x7fc0600a4dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc070000c00 tx=0x7fc070005fd0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0741014a0 msgr2=0x7fc07419d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:15.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0741014a0 0x7fc07419d650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc064009970 tx=0x7fc064009c80 comp rx=0 tx=0).stop 2024-03-16T11:03:15.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 shutdown_connections 2024-03-16T11:03:15.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc0600a2910 0x7fc0600a4dd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc074106580 0x7fc074197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0741014a0 0x7fc07419d650 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 --2- 172.21.15.78:0/1990063933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc074101e50 0x7fc07419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:15.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 >> 172.21.15.78:0/1990063933 conn(0x7fc074078300 msgr2=0x7fc0740ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:15.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 shutdown_connections 2024-03-16T11:03:15.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:15.726+0000 7fc07b5e7700 1 -- 172.21.15.78:0/1990063933 wait complete. 2024-03-16T11:03:16.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:16 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1990063933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:16.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:16 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1990063933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:16.495 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:03:16.496 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-16T11:03:16.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:16 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1990063933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:03:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:17 smithi098 ceph-mon[35848]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:17.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:17 smithi078 ceph-mon[32235]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:17 smithi153 ceph-mon[35521]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 -- 172.21.15.78:0/1684842568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 msgr2=0x7f05bc1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 --2- 172.21.15.78:0/1684842568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc1027a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f05b00099c0 tx=0x7f05b0009cd0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 -- 172.21.15.78:0/1684842568 shutdown_connections 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 --2- 172.21.15.78:0/1684842568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05bc102ce0 0x7f05bc10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 --2- 172.21.15.78:0/1684842568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05bc108300 0x7f05bc1086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 --2- 172.21.15.78:0/1684842568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc1027a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 -- 172.21.15.78:0/1684842568 >> 172.21.15.78:0/1684842568 conn(0x7f05bc079400 msgr2=0x7f05bc079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 -- 172.21.15.78:0/1684842568 shutdown_connections 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 -- 172.21.15.78:0/1684842568 wait complete. 2024-03-16T11:03:18.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.472+0000 7f05c2156700 1 Processor -- start 2024-03-16T11:03:18.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05c2156700 1 -- start start 2024-03-16T11:03:18.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05c2156700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:18.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05c2156700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05bc102ce0 0x7f05bc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:18.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05c2156700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05bc108300 0x7f05bc1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:18.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05c2156700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc112460 con 0x7f05bc102320 2024-03-16T11:03:18.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05c2156700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1122e0 con 0x7f05bc108300 2024-03-16T11:03:18.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05c2156700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1125e0 con 0x7f05bc102ce0 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60122/0 (socket says 172.21.15.78:60122) 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 -- 172.21.15.78:0/230784375 learned_addr learned my addr 172.21.15.78:0/230784375 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05baffd700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05bc102ce0 0x7f05bc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bbfff700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05bc108300 0x7f05bc1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 -- 172.21.15.78:0/230784375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05bc102ce0 msgr2=0x7f05bc19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05bc102ce0 0x7f05bc19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 -- 172.21.15.78:0/230784375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05bc108300 msgr2=0x7f05bc1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05bc108300 0x7f05bc1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 -- 172.21.15.78:0/230784375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05bc198fc0 con 0x7f05bc102320 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.473+0000 7f05bb7fe700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc19e660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00e9a0 tx=0x7f05ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.474+0000 7f05b8ff9700 1 -- 172.21.15.78:0/230784375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac00ca10 con 0x7f05bc102320 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.474+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05bc1992b0 con 0x7f05bc102320 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.474+0000 7f05b8ff9700 1 -- 172.21.15.78:0/230784375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05ac00cbb0 con 0x7f05bc102320 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.474+0000 7f05b8ff9700 1 -- 172.21.15.78:0/230784375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac018680 con 0x7f05bc102320 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.474+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05bc1a54c0 con 0x7f05bc102320 2024-03-16T11:03:18.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.475+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05bc1039b0 con 0x7f05bc102320 2024-03-16T11:03:18.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.476+0000 7f05b8ff9700 1 -- 172.21.15.78:0/230784375 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f05ac010430 con 0x7f05bc102320 2024-03-16T11:03:18.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.477+0000 7f05b8ff9700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f05a40a2890 0x7f05a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:18.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.477+0000 7f05baffd700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f05a40a2890 0x7f05a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:18.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.477+0000 7f05b8ff9700 1 -- 172.21.15.78:0/230784375 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f05ac0ce970 con 0x7f05bc102320 2024-03-16T11:03:18.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.478+0000 7f05baffd700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f05a40a2890 0x7f05a40a4d50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f05bc079ac0 tx=0x7f05b0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:18.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.480+0000 7f05b8ff9700 1 -- 172.21.15.78:0/230784375 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f05ac09be10 con 0x7f05bc102320 2024-03-16T11:03:18.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.674+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f05bc02db70 con 0x7f05bc102320 2024-03-16T11:03:18.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.675+0000 7f05b8ff9700 1 -- 172.21.15.78:0/230784375 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f05ac09b710 con 0x7f05bc102320 2024-03-16T11:03:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f05a40a2890 msgr2=0x7f05a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f05a40a2890 0x7f05a40a4d50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f05bc079ac0 tx=0x7f05b0005fb0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 msgr2=0x7f05bc19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc19e660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00e9a0 tx=0x7f05ac00ecb0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 shutdown_connections 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f05a40a2890 0x7f05a40a4d50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05bc108300 0x7f05bc1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc102320 0x7f05bc19e660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 --2- 172.21.15.78:0/230784375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05bc102ce0 0x7f05bc19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 >> 172.21.15.78:0/230784375 conn(0x7f05bc079400 msgr2=0x7f05bc1010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 shutdown_connections 2024-03-16T11:03:18.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:18.677+0000 7f05c2156700 1 -- 172.21.15.78:0/230784375 wait complete. 2024-03-16T11:03:19.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:19 smithi078 ceph-mon[32235]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:19.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:19 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/230784375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:19.309 INFO:tasks.cephadm:Adding mds.a on smithi078 2024-03-16T11:03:19.309 INFO:tasks.cephadm:Adding mds.d on smithi078 2024-03-16T11:03:19.309 INFO:tasks.cephadm:Adding mds.g on smithi078 2024-03-16T11:03:19.309 INFO:tasks.cephadm:Adding mds.j on smithi078 2024-03-16T11:03:19.309 INFO:tasks.cephadm:Adding mds.b on smithi098 2024-03-16T11:03:19.309 INFO:tasks.cephadm:Adding mds.e on smithi098 2024-03-16T11:03:19.309 INFO:tasks.cephadm:Adding mds.h on smithi098 2024-03-16T11:03:19.310 INFO:tasks.cephadm:Adding mds.k on smithi098 2024-03-16T11:03:19.310 INFO:tasks.cephadm:Adding mds.c on smithi153 2024-03-16T11:03:19.310 INFO:tasks.cephadm:Adding mds.f on smithi153 2024-03-16T11:03:19.310 INFO:tasks.cephadm:Adding mds.i on smithi153 2024-03-16T11:03:19.310 INFO:tasks.cephadm:Adding mds.l on smithi153 2024-03-16T11:03:19.310 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l' 2024-03-16T11:03:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:19 smithi098 ceph-mon[35848]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:19 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/230784375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:19 smithi153 ceph-mon[35521]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:19 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/230784375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:19.562 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/2150313679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102da0 msgr2=0x7fa1b4109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/2150313679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102da0 0x7fa1b4109650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00b0a0 tx=0x7fa1ac00b3b0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/2150313679 shutdown_connections 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/2150313679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102da0 0x7fa1b4109650 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/2150313679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1b4101200 0x7fa1b4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/2150313679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1b4102400 0x7fa1b4102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/2150313679 >> 172.21.15.153:0/2150313679 conn(0x7fa1b40fc7c0 msgr2=0x7fa1b40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/2150313679 shutdown_connections 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/2150313679 wait complete. 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 Processor -- start 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- start start 2024-03-16T11:03:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1b4101200 0x7fa1b4105140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102400 0x7fa1b41037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1b4102da0 0x7fa1b4103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410ccb0 con 0x7fa1b4102da0 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410cb30 con 0x7fa1b4101200 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.320+0000 7fa1bb8cb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410ce30 con 0x7fa1b4102400 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102400 0x7fa1b41037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102400 0x7fa1b41037e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59852/0 (socket says 172.21.15.153:59852) 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 -- 172.21.15.153:0/5917745 learned_addr learned my addr 172.21.15.153:0/5917745 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b9667700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1b4101200 0x7fa1b4105140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b9e68700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1b4102da0 0x7fa1b4103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 -- 172.21.15.153:0/5917745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1b4101200 msgr2=0x7fa1b4105140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1b4101200 0x7fa1b4105140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 -- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1b4102da0 msgr2=0x7fa1b4103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1b4102da0 0x7fa1b4103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 -- 172.21.15.153:0/5917745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b41045f0 con 0x7fa1b4102400 2024-03-16T11:03:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1b8e66700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102400 0x7fa1b41037e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400c970 tx=0x7fa1a400cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1aa7fc700 1 -- 172.21.15.153:0/5917745 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a40078d0 con 0x7fa1b4102400 2024-03-16T11:03:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1aa7fc700 1 -- 172.21.15.153:0/5917745 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1a4018480 con 0x7fa1b4102400 2024-03-16T11:03:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b41a25b0 con 0x7fa1b4102400 2024-03-16T11:03:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1aa7fc700 1 -- 172.21.15.153:0/5917745 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400f6b0 con 0x7fa1b4102400 2024-03-16T11:03:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.321+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b41a2ac0 con 0x7fa1b4102400 2024-03-16T11:03:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.322+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b4068f90 con 0x7fa1b4102400 2024-03-16T11:03:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.323+0000 7fa1aa7fc700 1 -- 172.21.15.153:0/5917745 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa1a4018630 con 0x7fa1b4102400 2024-03-16T11:03:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.324+0000 7fa1aa7fc700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa1a00a2910 0x7fa1a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.324+0000 7fa1aa7fc700 1 -- 172.21.15.153:0/5917745 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa1a40ce680 con 0x7fa1b4102400 2024-03-16T11:03:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.324+0000 7fa1b9667700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa1a00a2910 0x7fa1a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.325+0000 7fa1b9667700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa1a00a2910 0x7fa1a00a4dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4102260 tx=0x7fa1b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.328+0000 7fa1aa7fc700 1 -- 172.21.15.153:0/5917745 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa1a409bc50 con 0x7fa1b4102400 2024-03-16T11:03:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:21 smithi098 ceph-mon[35848]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.532+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}) -- 0x7fa1b40fea90 con 0x7fa1a00a2910 2024-03-16T11:03:21.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:21 smithi153 ceph-mon[35521]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:21.541 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mds.all update... 2024-03-16T11:03:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.541+0000 7fa1aa7fc700 1 -- 172.21.15.153:0/5917745 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fa1b40fea90 con 0x7fa1a00a2910 2024-03-16T11:03:21.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa1a00a2910 msgr2=0x7fa1a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:21.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa1a00a2910 0x7fa1a00a4dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4102260 tx=0x7fa1b000b410 comp rx=0 tx=0).stop 2024-03-16T11:03:21.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102400 msgr2=0x7fa1b41037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:21.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102400 0x7fa1b41037e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400c970 tx=0x7fa1a400cc80 comp rx=0 tx=0).stop 2024-03-16T11:03:21.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 shutdown_connections 2024-03-16T11:03:21.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa1a00a2910 0x7fa1a00a4dd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1b4102da0 0x7fa1b4103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1b4101200 0x7fa1b4105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 --2- 172.21.15.153:0/5917745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1b4102400 0x7fa1b41037e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 >> 172.21.15.153:0/5917745 conn(0x7fa1b40fc7c0 msgr2=0x7fa1b40fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 shutdown_connections 2024-03-16T11:03:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T11:03:21.543+0000 7fa1bb8cb700 1 -- 172.21.15.153:0/5917745 wait complete. 2024-03-16T11:03:21.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:21 smithi078 ceph-mon[32235]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:22.124 DEBUG:teuthology.orchestra.run.smithi078:mds.a> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.a.service 2024-03-16T11:03:22.126 DEBUG:teuthology.orchestra.run.smithi078:mds.d> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.d.service 2024-03-16T11:03:22.170 DEBUG:teuthology.orchestra.run.smithi078:mds.g> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.g.service 2024-03-16T11:03:22.215 DEBUG:teuthology.orchestra.run.smithi078:mds.j> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.j.service 2024-03-16T11:03:22.258 DEBUG:teuthology.orchestra.run.smithi098:mds.b> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.b.service 2024-03-16T11:03:22.261 DEBUG:teuthology.orchestra.run.smithi098:mds.e> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.e.service 2024-03-16T11:03:22.303 DEBUG:teuthology.orchestra.run.smithi098:mds.h> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.h.service 2024-03-16T11:03:22.347 DEBUG:teuthology.orchestra.run.smithi098:mds.k> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.k.service 2024-03-16T11:03:22.389 DEBUG:teuthology.orchestra.run.smithi153:mds.c> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.c.service 2024-03-16T11:03:22.392 DEBUG:teuthology.orchestra.run.smithi153:mds.f> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.f.service 2024-03-16T11:03:22.435 DEBUG:teuthology.orchestra.run.smithi153:mds.i> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.i.service 2024-03-16T11:03:22.478 DEBUG:teuthology.orchestra.run.smithi153:mds.l> sudo journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.l.service 2024-03-16T11:03:22.520 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-16T11:03:22.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:03:22.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- 172.21.15.78:0/1045879275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e4100700 msgr2=0x7f78e4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 --2- 172.21.15.78:0/1045879275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e4100700 0x7f78e4100b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009ca0 tx=0x7f78dc02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- 172.21.15.78:0/1045879275 shutdown_connections 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 --2- 172.21.15.78:0/1045879275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e41022b0 0x7f78e4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 --2- 172.21.15.78:0/1045879275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e41018f0 0x7f78e4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 --2- 172.21.15.78:0/1045879275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e4100700 0x7f78e4100b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- 172.21.15.78:0/1045879275 >> 172.21.15.78:0/1045879275 conn(0x7f78e40fbc70 msgr2=0x7f78e40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- 172.21.15.78:0/1045879275 shutdown_connections 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- 172.21.15.78:0/1045879275 wait complete. 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 Processor -- start 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- start start 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4100700 0x7f78e406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e41018f0 0x7f78e419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e41022b0 0x7f78e419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f78e410c220 con 0x7f78e41018f0 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78e410c0a0 con 0x7f78e4100700 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.633+0000 7f78eab1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78e410c3a0 con 0x7f78e41022b0 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e41018f0 0x7f78e419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e88b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4100700 0x7f78e406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e41018f0 0x7f78e419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60142/0 (socket says 172.21.15.78:60142) 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 -- 172.21.15.78:0/4262772767 learned_addr learned my addr 172.21.15.78:0/4262772767 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e90b8640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e41022b0 0x7f78e419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 -- 172.21.15.78:0/4262772767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e41022b0 msgr2=0x7f78e419ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e41022b0 0x7f78e419ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 -- 172.21.15.78:0/4262772767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4100700 msgr2=0x7f78e406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4100700 0x7f78e406a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 -- 172.21.15.78:0/4262772767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e419f6a0 con 0x7f78e41018f0 2024-03-16T11:03:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e3fff640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e41018f0 0x7f78e419c770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00e9f0 tx=0x7f78cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e1ffb640 1 -- 172.21.15.78:0/4262772767 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc00cd80 con 0x7f78e41018f0 2024-03-16T11:03:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e1ffb640 1 -- 172.21.15.78:0/4262772767 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78cc0054d0 con 0x7f78e41018f0 2024-03-16T11:03:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78e1ffb640 1 -- 172.21.15.78:0/4262772767 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc010490 con 0x7f78e41018f0 2024-03-16T11:03:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e419f930 con 0x7f78e41018f0 2024-03-16T11:03:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.634+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e419fe80 con 0x7f78e41018f0 2024-03-16T11:03:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.635+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78e4107a40 con 0x7f78e41018f0 2024-03-16T11:03:22.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.636+0000 7f78e1ffb640 1 -- 172.21.15.78:0/4262772767 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f78cc005020 con 0x7f78e41018f0 2024-03-16T11:03:22.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.637+0000 7f78e1ffb640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f78d00a2880 0x7f78d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.637+0000 7f78e88b7640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f78d00a2880 0x7f78d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.637+0000 7f78e1ffb640 1 -- 172.21.15.78:0/4262772767 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f78cc0ce8c0 con 0x7f78e41018f0 2024-03-16T11:03:22.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.637+0000 7f78e88b7640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f78d00a2880 0x7f78d00a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78dc032380 tx=0x7f78dc000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:22.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.641+0000 7f78e1ffb640 1 -- 172.21.15.78:0/4262772767 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f78cc09be00 con 0x7f78e41018f0 2024-03-16T11:03:22.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:03:22.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-03-16T11:03:22.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:22.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:22.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:22.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:22.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:22.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:22.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:22.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:22.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:22 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:22.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:22.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:22 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:22.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.819+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78e402d320 con 0x7f78e41018f0 2024-03-16T11:03:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.820+0000 7f78e1ffb640 1 -- 172.21.15.78:0/4262772767 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18373 (secure 0 0 0) 0x7f78cc09b700 con 0x7f78e41018f0 2024-03-16T11:03:22.821 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:22.821 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:03:10.113262+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:03:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f78d00a2880 msgr2=0x7f78d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f78d00a2880 0x7f78d00a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78dc032380 tx=0x7f78dc000f40 comp rx=0 tx=0).stop 2024-03-16T11:03:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e41018f0 msgr2=0x7f78e419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e41018f0 0x7f78e419c770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00e9f0 tx=0x7f78cc00edf0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 shutdown_connections 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f78d00a2880 0x7f78d00a4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e41022b0 0x7f78e419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e41018f0 0x7f78e419c770 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 --2- 172.21.15.78:0/4262772767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78e4100700 0x7f78e406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 >> 172.21.15.78:0/4262772767 conn(0x7f78e40fbc70 msgr2=0x7f78e4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 shutdown_connections 2024-03-16T11:03:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.823+0000 7f78eab1b640 1 -- 172.21.15.78:0/4262772767 wait complete. 2024-03-16T11:03:22.835 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:03:22.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:03:22.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:03:22.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:22.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:22 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:22.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- 172.21.15.78:0/4167821184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201029e0 msgr2=0x7fa620102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 --2- 172.21.15.78:0/4167821184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201029e0 0x7fa620102e60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa614009ca0 tx=0x7fa61402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- 172.21.15.78:0/4167821184 shutdown_connections 2024-03-16T11:03:22.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 --2- 172.21.15.78:0/4167821184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6201033a0 0x7fa620109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 --2- 172.21.15.78:0/4167821184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201029e0 0x7fa620102e60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 --2- 172.21.15.78:0/4167821184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6201017f0 0x7fa620101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- 172.21.15.78:0/4167821184 >> 172.21.15.78:0/4167821184 conn(0x7fa6200fcd80 msgr2=0x7fa6200ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- 172.21.15.78:0/4167821184 shutdown_connections 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- 172.21.15.78:0/4167821184 wait complete. 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 Processor -- start 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- start start 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6201017f0 0x7fa620075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6201029e0 0x7fa620073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201033a0 0x7fa620074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa62010d320 con 0x7fa6201033a0 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.979+0000 7fa6258f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa62010d1a0 con 0x7fa6201029e0 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa6258f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa62010d4a0 con 0x7fa6201017f0 2024-03-16T11:03:22.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201033a0 0x7fa620074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201033a0 0x7fa620074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60152/0 (socket says 172.21.15.78:60152) 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1069274445 learned_addr learned my addr 172.21.15.78:0/1069274445 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61e7fc640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6201029e0 0x7fa620073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61effd640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6201017f0 0x7fa620075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1069274445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6201017f0 msgr2=0x7fa620075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6201017f0 0x7fa620075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1069274445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6201029e0 msgr2=0x7fa620073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6201029e0 0x7fa620073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1069274445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6200749a0 con 0x7fa6201033a0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa61f7fe640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201033a0 0x7fa620074110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa61000d750 tx=0x7fa61000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa6248f5640 1 -- 172.21.15.78:0/1069274445 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610017070 con 0x7fa6201033a0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa6248f5640 1 -- 172.21.15.78:0/1069274445 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6100054d0 con 0x7fa6201033a0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6201a1b70 con 0x7fa6201033a0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa6248f5640 1 -- 172.21.15.78:0/1069274445 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610004c60 con 0x7fa6201033a0 2024-03-16T11:03:22.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.980+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6201a2090 con 0x7fa6201033a0 2024-03-16T11:03:22.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.981+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa620108cd0 con 0x7fa6201033a0 2024-03-16T11:03:22.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.982+0000 7fa6248f5640 1 -- 172.21.15.78:0/1069274445 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa610004e00 con 0x7fa6201033a0 2024-03-16T11:03:22.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.983+0000 7fa6248f5640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6080a2880 0x7fa6080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:22.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.983+0000 7fa61effd640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6080a2880 0x7fa6080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:22.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.983+0000 7fa6248f5640 1 -- 172.21.15.78:0/1069274445 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa6100cdf20 con 0x7fa6201033a0 2024-03-16T11:03:22.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.983+0000 7fa61effd640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6080a2880 0x7fa6080a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa618005fd0 tx=0x7fa618005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:22.987+0000 7fa6248f5640 1 -- 172.21.15.78:0/1069274445 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa61009b460 con 0x7fa6201033a0 2024-03-16T11:03:23.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.163+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa6200623e0 con 0x7fa6201033a0 2024-03-16T11:03:23.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.163+0000 7fa6248f5640 1 -- 172.21.15.78:0/1069274445 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7fa61009ad60 con 0x7fa6201033a0 2024-03-16T11:03:23.164 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6080a2880 msgr2=0x7fa6080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6080a2880 0x7fa6080a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa618005fd0 tx=0x7fa618005850 comp rx=0 tx=0).stop 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201033a0 msgr2=0x7fa620074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201033a0 0x7fa620074110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa61000d750 tx=0x7fa61000dc10 comp rx=0 tx=0).stop 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 shutdown_connections 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6201033a0 0x7fa620074110 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6201029e0 0x7fa620073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6080a2880 0x7fa6080a4d40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 --2- 172.21.15.78:0/1069274445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6201017f0 0x7fa620075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 >> 172.21.15.78:0/1069274445 conn(0x7fa6200fcd80 msgr2=0x7fa620105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 shutdown_connections 2024-03-16T11:03:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.166+0000 7fa6258f7640 1 -- 172.21.15.78:0/1069274445 wait complete. 2024-03-16T11:03:23.178 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:23.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 -- 172.21.15.78:0/355239089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901018f0 msgr2=0x7f6090101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:23.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 --2- 172.21.15.78:0/355239089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901018f0 0x7f6090101d70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6078009900 tx=0x7f607802bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:23.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 -- 172.21.15.78:0/355239089 shutdown_connections 2024-03-16T11:03:23.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 --2- 172.21.15.78:0/355239089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60901022b0 0x7f6090108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 --2- 172.21.15.78:0/355239089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901018f0 0x7f6090101d70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 --2- 172.21.15.78:0/355239089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6090100700 0x7f6090100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 -- 172.21.15.78:0/355239089 >> 172.21.15.78:0/355239089 conn(0x7f60900fbc70 msgr2=0x7f60900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 -- 172.21.15.78:0/355239089 shutdown_connections 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.284+0000 7f6095976640 1 -- 172.21.15.78:0/355239089 wait complete. 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 Processor -- start 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 -- start start 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6090100700 0x7f6090196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60901018f0 0x7f6090196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901022b0 0x7f609019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f609010c210 con 0x7f60901022b0 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f609010c090 con 0x7f6090100700 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f6095976640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f609010c390 con 0x7f60901018f0 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f608f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901022b0 0x7f609019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f608f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901022b0 0x7f609019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60172/0 (socket says 172.21.15.78:60172) 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f608f7fe640 1 -- 172.21.15.78:0/1646010047 learned_addr learned my addr 172.21.15.78:0/1646010047 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f608e7fc640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60901018f0 0x7f6090196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.285+0000 7f608effd640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6090100700 0x7f6090196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f608f7fe640 1 -- 172.21.15.78:0/1646010047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60901018f0 msgr2=0x7f6090196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f608f7fe640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60901018f0 0x7f6090196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f608f7fe640 1 -- 172.21.15.78:0/1646010047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6090100700 msgr2=0x7f6090196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f608f7fe640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6090100700 0x7f6090196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f608f7fe640 1 -- 172.21.15.78:0/1646010047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609019e550 con 0x7f60901022b0 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f608f7fe640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901022b0 0x7f609019dee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f608000e970 tx=0x7f608000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f6094974640 1 -- 172.21.15.78:0/1646010047 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608000ccc0 con 0x7f60901022b0 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f6094974640 1 -- 172.21.15.78:0/1646010047 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f608000ce60 con 0x7f60901022b0 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609019e840 con 0x7f60901022b0 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f6094974640 1 -- 172.21.15.78:0/1646010047 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080010430 con 0x7f60901022b0 2024-03-16T11:03:23.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.286+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f609019ed60 con 0x7f60901022b0 2024-03-16T11:03:23.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.287+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6090107af0 con 0x7f60901022b0 2024-03-16T11:03:23.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.288+0000 7f6094974640 1 -- 172.21.15.78:0/1646010047 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6080004ff0 con 0x7f60901022b0 2024-03-16T11:03:23.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.289+0000 7f6094974640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f607c0a2880 0x7f607c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:23.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.289+0000 7f608effd640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f607c0a2880 0x7f607c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:23.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.289+0000 7f6094974640 1 -- 172.21.15.78:0/1646010047 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f60800ce950 con 0x7f60901022b0 2024-03-16T11:03:23.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.289+0000 7f608effd640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f607c0a2880 0x7f607c0a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6090101750 tx=0x7f6088009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:23.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.293+0000 7f6094974640 1 -- 172.21.15.78:0/1646010047 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f608009be90 con 0x7f60901022b0 2024-03-16T11:03:23.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.517+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f609002d3b0 con 0x7f60901022b0 2024-03-16T11:03:23.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.517+0000 7f6094974640 1 -- 172.21.15.78:0/1646010047 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f608009b790 con 0x7f60901022b0 2024-03-16T11:03:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:23.518 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T11:03:23.519 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 1 2024-03-16T11:03:23.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f607c0a2880 msgr2=0x7f607c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:23.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f607c0a2880 0x7f607c0a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6090101750 tx=0x7f6088009670 comp rx=0 tx=0).stop 2024-03-16T11:03:23.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901022b0 msgr2=0x7f609019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901022b0 0x7f609019dee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f608000e970 tx=0x7f608000ed70 comp rx=0 tx=0).stop 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 shutdown_connections 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60901022b0 0x7f609019dee0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f607c0a2880 0x7f607c0a4d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60901018f0 0x7f6090196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 --2- 172.21.15.78:0/1646010047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6090100700 0x7f6090196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 >> 172.21.15.78:0/1646010047 conn(0x7f60900fbc70 msgr2=0x7f6090104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 shutdown_connections 2024-03-16T11:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:23.520+0000 7f6095976640 1 -- 172.21.15.78:0/1646010047 wait complete. 2024-03-16T11:03:23.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:23 smithi078 ceph-mon[32235]: Deploying daemon mds.l on smithi153 2024-03-16T11:03:23.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:23 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4262772767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:23.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:23 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1069274445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:23.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:23 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1646010047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:23.846 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:23 smithi153 ceph-mon[35521]: Deploying daemon mds.l on smithi153 2024-03-16T11:03:23.846 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:23 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4262772767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:23.846 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:23 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1069274445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:23.846 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:23 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1646010047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:23.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:23 smithi098 ceph-mon[35848]: Deploying daemon mds.l on smithi153 2024-03-16T11:03:23.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:23 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4262772767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:23.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:23 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1069274445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:23.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:23 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1646010047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:24.543 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:03:24 smithi153 podman[66153]: 2024-03-16 11:03:24.440347001 +0000 UTC m=+0.120547624 container create 9455dd4412f1f247277bdcc292671e22dd8b57725e1d3da708a8759434777c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD) 2024-03-16T11:03:24.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:24 smithi078 ceph-mon[32235]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:24.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:24 smithi098 ceph-mon[35848]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:25.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:24 smithi153 ceph-mon[35521]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:25.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:03:24 smithi153 podman[66153]: 2024-03-16 11:03:24.659755307 +0000 UTC m=+0.339955893 container init 9455dd4412f1f247277bdcc292671e22dd8b57725e1d3da708a8759434777c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:03:25.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:03:24 smithi153 podman[66153]: 2024-03-16 11:03:24.662333636 +0000 UTC m=+0.342534217 container start 9455dd4412f1f247277bdcc292671e22dd8b57725e1d3da708a8759434777c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T11:03:25.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:03:24 smithi153 ceph-mds[66172]: -- 172.21.15.153:0/620660728 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a1f6983a40 con 0x55a1f6956800 2024-03-16T11:03:25.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:03:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l[66168]: starting mds.l at 2024-03-16T11:03:25.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:03:24 smithi153 bash[66153]: 9455dd4412f1f247277bdcc292671e22dd8b57725e1d3da708a8759434777c3e 2024-03-16T11:03:25.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:03:24 smithi153 systemd[1]: Started Ceph mds.l for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:25.533 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/2947420920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec100700 msgr2=0x7f0fec100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/2947420920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec100700 0x7f0fec100b20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009900 tx=0x7f0fdc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/2947420920 shutdown_connections 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/2947420920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fec1022b0 0x7f0fec108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/2947420920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fec1018f0 0x7f0fec101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/2947420920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec100700 0x7f0fec100b20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/2947420920 >> 172.21.15.78:0/2947420920 conn(0x7f0fec0fbc70 msgr2=0x7f0fec0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/2947420920 shutdown_connections 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/2947420920 wait complete. 2024-03-16T11:03:25.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 Processor -- start 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.651+0000 7f0ff3ff4640 1 -- start start 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fec100700 0x7f0fec069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec1018f0 0x7f0fec19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fec1022b0 0x7f0fec19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec10c200 con 0x7f0fec1018f0 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec10c080 con 0x7f0fec1022b0 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec10c380 con 0x7f0fec100700 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec1018f0 0x7f0fec19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec1018f0 0x7f0fec19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36484/0 (socket says 172.21.15.78:36484) 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 -- 172.21.15.78:0/1838543814 learned_addr learned my addr 172.21.15.78:0/1838543814 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff2591640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fec1022b0 0x7f0fec19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff1d90640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fec100700 0x7f0fec069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 -- 172.21.15.78:0/1838543814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fec100700 msgr2=0x7f0fec069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:25.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fec100700 0x7f0fec069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 -- 172.21.15.78:0/1838543814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fec1022b0 msgr2=0x7f0fec19ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fec1022b0 0x7f0fec19ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 -- 172.21.15.78:0/1838543814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fec19f620 con 0x7f0fec1018f0 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff158f640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec1018f0 0x7f0fec19c6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800e9f0 tx=0x7f0fe800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0fe2ffd640 1 -- 172.21.15.78:0/1838543814 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800cd80 con 0x7f0fec1018f0 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fec19f910 con 0x7f0fec1018f0 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0fe2ffd640 1 -- 172.21.15.78:0/1838543814 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fe80108e0 con 0x7f0fec1018f0 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0fe2ffd640 1 -- 172.21.15.78:0/1838543814 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8018920 con 0x7f0fec1018f0 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.652+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fec19fe60 con 0x7f0fec1018f0 2024-03-16T11:03:25.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.653+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fec107aa0 con 0x7f0fec1018f0 2024-03-16T11:03:25.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.654+0000 7f0fe2ffd640 1 -- 172.21.15.78:0/1838543814 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0fe8010a90 con 0x7f0fec1018f0 2024-03-16T11:03:25.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.655+0000 7f0fe2ffd640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0fd80a2800 0x7f0fd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:25.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.655+0000 7f0ff1d90640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0fd80a2800 0x7f0fd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:25.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.655+0000 7f0fe2ffd640 1 -- 172.21.15.78:0/1838543814 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f0fe80ced70 con 0x7f0fec1018f0 2024-03-16T11:03:25.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.656+0000 7f0ff1d90640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0fd80a2800 0x7f0fd80a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc005ab0 tx=0x7f0fdc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.659+0000 7f0fe2ffd640 1 -- 172.21.15.78:0/1838543814 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0fe809bc40 con 0x7f0fec1018f0 2024-03-16T11:03:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.871+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fec02d1b0 con 0x7f0fec1018f0 2024-03-16T11:03:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.871+0000 7f0fe2ffd640 1 -- 172.21.15.78:0/1838543814 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f0fe809fce0 con 0x7f0fec1018f0 2024-03-16T11:03:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:25.872 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:03:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0fd80a2800 msgr2=0x7f0fd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0fd80a2800 0x7f0fd80a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc005ab0 tx=0x7f0fdc005fd0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec1018f0 msgr2=0x7f0fec19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec1018f0 0x7f0fec19c6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800e9f0 tx=0x7f0fe800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 shutdown_connections 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fec1022b0 0x7f0fec19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fec1018f0 0x7f0fec19c6f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0fd80a2800 0x7f0fd80a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 --2- 172.21.15.78:0/1838543814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fec100700 0x7f0fec069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 >> 172.21.15.78:0/1838543814 conn(0x7f0fec0fbc70 msgr2=0x7f0fec104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 shutdown_connections 2024-03-16T11:03:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:25.873+0000 7f0ff3ff4640 1 -- 172.21.15.78:0/1838543814 wait complete. 2024-03-16T11:03:25.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6832/3909734389,v1:172.21.15.153:6833/3909734389] up:boot 2024-03-16T11:03:25.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: fsmap 1 up:standby 2024-03-16T11:03:25.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:03:25.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:25.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:25.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:25.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:25.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:25.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:25.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:25 smithi078 ceph-mon[32235]: Deploying daemon mds.k on smithi098 2024-03-16T11:03:25.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6832/3909734389,v1:172.21.15.153:6833/3909734389] up:boot 2024-03-16T11:03:25.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: fsmap 1 up:standby 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:25.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:25 smithi098 ceph-mon[35848]: Deploying daemon mds.k on smithi098 2024-03-16T11:03:26.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6832/3909734389,v1:172.21.15.153:6833/3909734389] up:boot 2024-03-16T11:03:26.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: fsmap 1 up:standby 2024-03-16T11:03:26.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:03:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:25 smithi153 ceph-mon[35521]: Deploying daemon mds.k on smithi098 2024-03-16T11:03:26.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:26 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1838543814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:26.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:26 smithi098 ceph-mon[35848]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:26.965 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:26 smithi098 systemd[1]: Starting Ceph mds.k for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:27.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:26 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1838543814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:27.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:26 smithi153 ceph-mon[35521]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:27.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:26 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1838543814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:27.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:26 smithi078 ceph-mon[32235]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:27.464 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 podman[66048]: 2024-03-16 11:03:27.006640118 +0000 UTC m=+0.018225796 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:27.465 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 podman[66048]: 2024-03-16 11:03:27.135746929 +0000 UTC m=+0.147332599 container create c3d1d4e297309b26ff87396a7aca659fee0cf9dd81dd46d19962f57941fe61c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-16T11:03:27.465 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 podman[66048]: 2024-03-16 11:03:27.46442833 +0000 UTC m=+0.476014014 container init c3d1d4e297309b26ff87396a7aca659fee0cf9dd81dd46d19962f57941fe61c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-16T11:03:27.714 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 podman[66048]: 2024-03-16 11:03:27.467425849 +0000 UTC m=+0.479011524 container start c3d1d4e297309b26ff87396a7aca659fee0cf9dd81dd46d19962f57941fe61c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T11:03:27.714 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 ceph-mds[66067]: -- 172.21.15.98:0/2621473230 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5640253fba40 con 0x5640253ce800 2024-03-16T11:03:27.715 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k[66063]: starting mds.k at 2024-03-16T11:03:27.715 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 bash[66048]: c3d1d4e297309b26ff87396a7aca659fee0cf9dd81dd46d19962f57941fe61c5 2024-03-16T11:03:27.715 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:03:27 smithi098 systemd[1]: Started Ceph mds.k for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:27.889 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:27.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 -- 172.21.15.78:0/1135642376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 msgr2=0x7faa14101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:27.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 --2- 172.21.15.78:0/1135642376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14101d70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009900 tx=0x7fa9fc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:27.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 -- 172.21.15.78:0/1135642376 shutdown_connections 2024-03-16T11:03:27.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 --2- 172.21.15.78:0/1135642376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa141022b0 0x7faa14108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:27.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 --2- 172.21.15.78:0/1135642376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14101d70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:27.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 --2- 172.21.15.78:0/1135642376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14100700 0x7faa14100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:27.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 -- 172.21.15.78:0/1135642376 >> 172.21.15.78:0/1135642376 conn(0x7faa140fbc70 msgr2=0x7faa140fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.992+0000 7faa19513640 1 -- 172.21.15.78:0/1135642376 shutdown_connections 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 -- 172.21.15.78:0/1135642376 wait complete. 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 Processor -- start 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 -- start start 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa14100700 0x7faa14196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa141022b0 0x7faa1419df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7faa1410c280 con 0x7faa141018f0 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa1410c100 con 0x7faa141022b0 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa19513640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faa1410c400 con 0x7faa14100700 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa0bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa0bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36500/0 (socket says 172.21.15.78:36500) 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa0bfff640 1 -- 172.21.15.78:0/3945654520 learned_addr learned my addr 172.21.15.78:0/3945654520 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:27.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.993+0000 7faa137fe640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa141022b0 0x7faa1419df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa12ffd640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa14100700 0x7faa14196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa0bfff640 1 -- 172.21.15.78:0/3945654520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa14100700 msgr2=0x7faa14196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa0bfff640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa14100700 0x7faa14196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa0bfff640 1 -- 172.21.15.78:0/3945654520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa141022b0 msgr2=0x7faa1419df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa0bfff640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa141022b0 0x7faa1419df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa0bfff640 1 -- 172.21.15.78:0/3945654520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1419e580 con 0x7faa141018f0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa0bfff640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14196ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc000c00 tx=0x7fa9fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa10ff9640 1 -- 172.21.15.78:0/3945654520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc043020 con 0x7faa141018f0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa1419e810 con 0x7faa141018f0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa10ff9640 1 -- 172.21.15.78:0/3945654520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9fc0054c0 con 0x7faa141018f0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa10ff9640 1 -- 172.21.15.78:0/3945654520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc003d70 con 0x7faa141018f0 2024-03-16T11:03:27.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.994+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa1419ed60 con 0x7faa141018f0 2024-03-16T11:03:27.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.995+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa14107a40 con 0x7faa141018f0 2024-03-16T11:03:27.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.996+0000 7faa10ff9640 1 -- 172.21.15.78:0/3945654520 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa9fc046020 con 0x7faa141018f0 2024-03-16T11:03:27.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.997+0000 7faa10ff9640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa000a2800 0x7faa000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:27.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.997+0000 7faa12ffd640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa000a2800 0x7faa000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.997+0000 7faa10ff9640 1 -- 172.21.15.78:0/3945654520 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa9fc0f0c50 con 0x7faa141018f0 2024-03-16T11:03:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:27.997+0000 7faa12ffd640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa000a2800 0x7faa000a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faa04005fd0 tx=0x7faa04005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:28.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.005+0000 7faa10ff9640 1 -- 172.21.15.78:0/3945654520 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa9fc0f6020 con 0x7faa141018f0 2024-03-16T11:03:28.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.240+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa1402d3d0 con 0x7faa141018f0 2024-03-16T11:03:28.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.240+0000 7faa10ff9640 1 -- 172.21.15.78:0/3945654520 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fa9fc04d020 con 0x7faa141018f0 2024-03-16T11:03:28.241 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:28.241 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:03:28.242 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2024-03-16T11:03:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.242+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa000a2800 msgr2=0x7faa000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.242+0000 7faa19513640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa000a2800 0x7faa000a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faa04005fd0 tx=0x7faa04005850 comp rx=0 tx=0).stop 2024-03-16T11:03:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.242+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 msgr2=0x7faa14196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.242+0000 7faa19513640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14196ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc000c00 tx=0x7fa9fc0042f0 comp rx=0 tx=0).stop 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 shutdown_connections 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa141022b0 0x7faa1419df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa141018f0 0x7faa14196ea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa000a2800 0x7faa000a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 --2- 172.21.15.78:0/3945654520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa14100700 0x7faa14196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 >> 172.21.15.78:0/3945654520 conn(0x7faa140fbc70 msgr2=0x7faa14104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 shutdown_connections 2024-03-16T11:03:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:28.243+0000 7faa19513640 1 -- 172.21.15.78:0/3945654520 wait complete. 2024-03-16T11:03:28.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:28.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:28.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:28.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: Deploying daemon mds.j on smithi078 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6832/3200006820,v1:172.21.15.98:6833/3200006820] up:boot 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: fsmap 2 up:standby 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:28.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:28 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3945654520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: Deploying daemon mds.j on smithi078 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6832/3200006820,v1:172.21.15.98:6833/3200006820] up:boot 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: fsmap 2 up:standby 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:03:28.856 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:28.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:28 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3945654520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:29.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:29.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:29.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:29.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: Deploying daemon mds.j on smithi078 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6832/3200006820,v1:172.21.15.98:6833/3200006820] up:boot 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: fsmap 2 up:standby 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:28 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3945654520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:29.992 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:29 smithi078 systemd[1]: Starting Ceph mds.j for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:30.244 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:29 smithi078 podman[73231]: 2024-03-16 11:03:29.991919011 +0000 UTC m=+0.109762551 container create e69bad745878ee0910592101bbfa330aa4bcf9c96367f3bd654a629113874f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:03:30.244 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:29 smithi078 podman[73231]: 2024-03-16 11:03:29.901026252 +0000 UTC m=+0.018869800 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:30.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 -- 172.21.15.78:0/930265488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca541022b0 msgr2=0x7fca54108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 --2- 172.21.15.78:0/930265488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca541022b0 0x7fca54108b40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009ca0 tx=0x7fca4c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 -- 172.21.15.78:0/930265488 shutdown_connections 2024-03-16T11:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 --2- 172.21.15.78:0/930265488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca541022b0 0x7fca54108b40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 --2- 172.21.15.78:0/930265488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca541018f0 0x7fca54101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 --2- 172.21.15.78:0/930265488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca54100700 0x7fca54100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 -- 172.21.15.78:0/930265488 >> 172.21.15.78:0/930265488 conn(0x7fca540fbc70 msgr2=0x7fca540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 -- 172.21.15.78:0/930265488 shutdown_connections 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.364+0000 7fca5af74640 1 -- 172.21.15.78:0/930265488 wait complete. 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 Processor -- start 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 -- start start 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca54100700 0x7fca540694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca541018f0 0x7fca5419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca541022b0 0x7fca5419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fca5410c230 con 0x7fca54100700 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca5410c0b0 con 0x7fca541022b0 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca5af74640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fca5410c3b0 con 0x7fca541018f0 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.365+0000 7fca58d10640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca54100700 0x7fca540694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca54100700 0x7fca540694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36538/0 (socket says 172.21.15.78:36538) 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 -- 172.21.15.78:0/2643781289 learned_addr learned my addr 172.21.15.78:0/2643781289 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca59511640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca541022b0 0x7fca5419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 -- 172.21.15.78:0/2643781289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca541018f0 msgr2=0x7fca5419c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca541018f0 0x7fca5419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 -- 172.21.15.78:0/2643781289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca541022b0 msgr2=0x7fca5419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca541022b0 0x7fca5419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 -- 172.21.15.78:0/2643781289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5419f650 con 0x7fca54100700 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca59511640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca541022b0 0x7fca5419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca58d10640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca54100700 0x7fca540694b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca4800d950 tx=0x7fca4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca51ffb640 1 -- 172.21.15.78:0/2643781289 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48017070 con 0x7fca54100700 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5419f8e0 con 0x7fca54100700 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca51ffb640 1 -- 172.21.15.78:0/2643781289 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca48009c30 con 0x7fca54100700 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca51ffb640 1 -- 172.21.15.78:0/2643781289 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4800f620 con 0x7fca54100700 2024-03-16T11:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.366+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca5419fe30 con 0x7fca54100700 2024-03-16T11:03:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.368+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca54107aa0 con 0x7fca54100700 2024-03-16T11:03:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.369+0000 7fca51ffb640 1 -- 172.21.15.78:0/2643781289 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fca4801a070 con 0x7fca54100700 2024-03-16T11:03:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.369+0000 7fca51ffb640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fca440a2800 0x7fca440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:30.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.369+0000 7fca53fff640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fca440a2800 0x7fca440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:30.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.370+0000 7fca51ffb640 1 -- 172.21.15.78:0/2643781289 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fca480ce0a0 con 0x7fca54100700 2024-03-16T11:03:30.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.370+0000 7fca53fff640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fca440a2800 0x7fca440a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca400058c0 tx=0x7fca40005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:30.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.374+0000 7fca51ffb640 1 -- 172.21.15.78:0/2643781289 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca48009de0 con 0x7fca54100700 2024-03-16T11:03:30.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:30 smithi078 podman[73231]: 2024-03-16 11:03:30.244446976 +0000 UTC m=+0.362290515 container init e69bad745878ee0910592101bbfa330aa4bcf9c96367f3bd654a629113874f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-16T11:03:30.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:30 smithi078 podman[73231]: 2024-03-16 11:03:30.24723505 +0000 UTC m=+0.365078579 container start e69bad745878ee0910592101bbfa330aa4bcf9c96367f3bd654a629113874f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T11:03:30.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:30 smithi078 ceph-mds[73250]: -- 172.21.15.78:0/2149872075 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55850b515a40 con 0x55850b4e8800 2024-03-16T11:03:30.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j[73246]: starting mds.j at 2024-03-16T11:03:30.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:30 smithi078 bash[73231]: e69bad745878ee0910592101bbfa330aa4bcf9c96367f3bd654a629113874f35 2024-03-16T11:03:30.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:03:30 smithi078 systemd[1]: Started Ceph mds.j for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.625+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca5402d1b0 con 0x7fca54100700 2024-03-16T11:03:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.626+0000 7fca51ffb640 1 -- 172.21.15.78:0/2643781289 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fca4809b5e0 con 0x7fca54100700 2024-03-16T11:03:30.627 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:30.627 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:03:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fca440a2800 msgr2=0x7fca440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fca440a2800 0x7fca440a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca400058c0 tx=0x7fca40005850 comp rx=0 tx=0).stop 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca54100700 msgr2=0x7fca540694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca54100700 0x7fca540694b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca4800d950 tx=0x7fca4800de10 comp rx=0 tx=0).stop 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 shutdown_connections 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca541022b0 0x7fca5419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fca440a2800 0x7fca440a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca541018f0 0x7fca5419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 --2- 172.21.15.78:0/2643781289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca54100700 0x7fca540694b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 >> 172.21.15.78:0/2643781289 conn(0x7fca540fbc70 msgr2=0x7fca54104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 shutdown_connections 2024-03-16T11:03:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:30.628+0000 7fca5af74640 1 -- 172.21.15.78:0/2643781289 wait complete. 2024-03-16T11:03:31.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:31.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:31 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2643781289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:31.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:31.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:31.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:31.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:31.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:31 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2643781289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:31 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2643781289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:32 smithi153 ceph-mon[35521]: Deploying daemon mds.d on smithi078 2024-03-16T11:03:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:32 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.78:6834/1233514200,v1:172.21.15.78:6835/1233514200] up:boot 2024-03-16T11:03:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:32 smithi153 ceph-mon[35521]: fsmap 3 up:standby 2024-03-16T11:03:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:32 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:03:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:32 smithi078 ceph-mon[32235]: Deploying daemon mds.d on smithi078 2024-03-16T11:03:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:32 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.78:6834/1233514200,v1:172.21.15.78:6835/1233514200] up:boot 2024-03-16T11:03:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:32 smithi078 ceph-mon[32235]: fsmap 3 up:standby 2024-03-16T11:03:32.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:32 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:03:32.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:32 smithi098 ceph-mon[35848]: Deploying daemon mds.d on smithi078 2024-03-16T11:03:32.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:32 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.78:6834/1233514200,v1:172.21.15.78:6835/1233514200] up:boot 2024-03-16T11:03:32.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:32 smithi098 ceph-mon[35848]: fsmap 3 up:standby 2024-03-16T11:03:32.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:32 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:03:32.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:32.670 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:32 smithi078 systemd[1]: Starting Ceph mds.d for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:32.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 -- 172.21.15.78:0/3047511628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 msgr2=0x7faf08108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/3047511628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf08108b40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf00009ca0 tx=0x7faf0002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 -- 172.21.15.78:0/3047511628 shutdown_connections 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/3047511628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf08108b40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/3047511628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf081018f0 0x7faf08101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/3047511628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf08100700 0x7faf08100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 -- 172.21.15.78:0/3047511628 >> 172.21.15.78:0/3047511628 conn(0x7faf080fbc70 msgr2=0x7faf080fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 -- 172.21.15.78:0/3047511628 shutdown_connections 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 -- 172.21.15.78:0/3047511628 wait complete. 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 Processor -- start 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.751+0000 7faf0e6f1640 1 -- start start 2024-03-16T11:03:32.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf08100700 0x7faf080694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf081018f0 0x7faf0819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf0819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810c210 con 0x7faf081022b0 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810c090 con 0x7faf081018f0 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810c390 con 0x7faf08100700 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf0819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf0819eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36564/0 (socket says 172.21.15.78:36564) 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 -- 172.21.15.78:0/598746334 learned_addr learned my addr 172.21.15.78:0/598746334 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf077fe640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf081018f0 0x7faf0819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf07fff640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf08100700 0x7faf080694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 -- 172.21.15.78:0/598746334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf08100700 msgr2=0x7faf080694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf08100700 0x7faf080694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 -- 172.21.15.78:0/598746334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf081018f0 msgr2=0x7faf0819c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf081018f0 0x7faf0819c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 -- 172.21.15.78:0/598746334 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0819f5b0 con 0x7faf081022b0 2024-03-16T11:03:32.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0cc8e640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf0819eca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf00032380 tx=0x7faf00032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:32.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf057fa640 1 -- 172.21.15.78:0/598746334 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00041020 con 0x7faf081022b0 2024-03-16T11:03:32.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf057fa640 1 -- 172.21.15.78:0/598746334 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf00003e20 con 0x7faf081022b0 2024-03-16T11:03:32.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0819f7e0 con 0x7faf081022b0 2024-03-16T11:03:32.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf057fa640 1 -- 172.21.15.78:0/598746334 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00039040 con 0x7faf081022b0 2024-03-16T11:03:32.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.752+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0819fd20 con 0x7faf081022b0 2024-03-16T11:03:32.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.754+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf081079c0 con 0x7faf081022b0 2024-03-16T11:03:32.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.755+0000 7faf057fa640 1 -- 172.21.15.78:0/598746334 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7faf00044020 con 0x7faf081022b0 2024-03-16T11:03:32.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.755+0000 7faf057fa640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faef00a2880 0x7faef00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:32.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.755+0000 7faf07fff640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faef00a2880 0x7faef00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:32.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.755+0000 7faf057fa640 1 -- 172.21.15.78:0/598746334 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7faf000f07c0 con 0x7faf081022b0 2024-03-16T11:03:32.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.756+0000 7faf07fff640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faef00a2880 0x7faef00a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faef8005fd0 tx=0x7faef8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:32.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.760+0000 7faf057fa640 1 -- 172.21.15.78:0/598746334 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf0004b020 con 0x7faf081022b0 2024-03-16T11:03:32.921 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:32 smithi078 podman[73588]: 2024-03-16 11:03:32.675947565 +0000 UTC m=+0.018353695 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:32.921 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:32 smithi078 podman[73588]: 2024-03-16 11:03:32.794900151 +0000 UTC m=+0.137306272 container create 721c557738ef6dedcdf944c6fb4b49e24354d0366a847db51c9530f61a498693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:03:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.981+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf0802d3d0 con 0x7faf081022b0 2024-03-16T11:03:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.982+0000 7faf057fa640 1 -- 172.21.15.78:0/598746334 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7faf000bdd00 con 0x7faf081022b0 2024-03-16T11:03:32.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:03:32.983 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 4 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faef00a2880 msgr2=0x7faef00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faef00a2880 0x7faef00a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faef8005fd0 tx=0x7faef8005850 comp rx=0 tx=0).stop 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 msgr2=0x7faf0819eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf0819eca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf00032380 tx=0x7faf00032820 comp rx=0 tx=0).stop 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 shutdown_connections 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf081022b0 0x7faf0819eca0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf081018f0 0x7faf0819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faef00a2880 0x7faef00a4d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 --2- 172.21.15.78:0/598746334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf08100700 0x7faf080694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 >> 172.21.15.78:0/598746334 conn(0x7faf080fbc70 msgr2=0x7faf08104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 shutdown_connections 2024-03-16T11:03:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:32.984+0000 7faf0e6f1640 1 -- 172.21.15.78:0/598746334 wait complete. 2024-03-16T11:03:33.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:33 smithi153 ceph-mon[35521]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:33.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:33 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/598746334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:33 smithi078 ceph-mon[32235]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:33 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/598746334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:33 smithi078 podman[73588]: 2024-03-16 11:03:33.039164439 +0000 UTC m=+0.381570574 container init 721c557738ef6dedcdf944c6fb4b49e24354d0366a847db51c9530f61a498693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:33 smithi078 podman[73588]: 2024-03-16 11:03:33.041927621 +0000 UTC m=+0.384333746 container start 721c557738ef6dedcdf944c6fb4b49e24354d0366a847db51c9530f61a498693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:33 smithi078 ceph-mds[73640]: -- 172.21.15.78:0/7769320 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec79a77a40 con 0x55ec79a4a800 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:33 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d[73634]: starting mds.d at 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:33 smithi078 bash[73588]: 721c557738ef6dedcdf944c6fb4b49e24354d0366a847db51c9530f61a498693 2024-03-16T11:03:33.309 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:03:33 smithi078 systemd[1]: Started Ceph mds.d for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:33.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:33 smithi098 ceph-mon[35848]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:33.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:33 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/598746334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.78:6836/4216287385,v1:172.21.15.78:6837/4216287385] up:boot 2024-03-16T11:03:34.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: fsmap 4 up:standby 2024-03-16T11:03:34.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:34 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:03:34.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.78:6836/4216287385,v1:172.21.15.78:6837/4216287385] up:boot 2024-03-16T11:03:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: fsmap 4 up:standby 2024-03-16T11:03:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:34 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:03:34.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:34.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:34.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:34.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:34.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.78:6836/4216287385,v1:172.21.15.78:6837/4216287385] up:boot 2024-03-16T11:03:34.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: fsmap 4 up:standby 2024-03-16T11:03:34.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:34 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:03:34.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/4032205194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 msgr2=0x7fa3b41046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/4032205194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b41046d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009900 tx=0x7fa3a802bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/4032205194 shutdown_connections 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/4032205194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b4107540 0x7fa3b4109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/4032205194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4104c10 0x7fa3b4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/4032205194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b41046d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/4032205194 >> 172.21.15.78:0/4032205194 conn(0x7fa3b40fbc70 msgr2=0x7fa3b40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/4032205194 shutdown_connections 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/4032205194 wait complete. 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 Processor -- start 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.109+0000 7fa3b8ba9640 1 -- start start 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b4104c10 0x7fa3b41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4107540 0x7fa3b419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b410d420 con 0x7fa3b41022e0 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b410d2a0 con 0x7fa3b4104c10 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b410d5a0 con 0x7fa3b4107540 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b4197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36608/0 (socket says 172.21.15.78:36608) 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 -- 172.21.15.78:0/3206564613 learned_addr learned my addr 172.21.15.78:0/3206564613 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b1d9b640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b4104c10 0x7fa3b41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 -- 172.21.15.78:0/3206564613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4107540 msgr2=0x7fa3b419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4107540 0x7fa3b419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 -- 172.21.15.78:0/3206564613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b4104c10 msgr2=0x7fa3b41980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b4104c10 0x7fa3b41980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 -- 172.21.15.78:0/3206564613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b419f7b0 con 0x7fa3b41022e0 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b259c640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b4197b90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8036040 tx=0x7fa3a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3a37fe640 1 -- 172.21.15.78:0/3206564613 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8043020 con 0x7fa3b41022e0 2024-03-16T11:03:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3a37fe640 1 -- 172.21.15.78:0/3206564613 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3a80054c0 con 0x7fa3b41022e0 2024-03-16T11:03:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3a37fe640 1 -- 172.21.15.78:0/3206564613 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8003d70 con 0x7fa3b41022e0 2024-03-16T11:03:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b419fa40 con 0x7fa3b41022e0 2024-03-16T11:03:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.110+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b41a0060 con 0x7fa3b41022e0 2024-03-16T11:03:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.112+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3b40fd760 con 0x7fa3b41022e0 2024-03-16T11:03:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.112+0000 7fa3a37fe640 1 -- 172.21.15.78:0/3206564613 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa3a8046020 con 0x7fa3b41022e0 2024-03-16T11:03:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.113+0000 7fa3a37fe640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa39c0a2880 0x7fa39c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.113+0000 7fa3b1d9b640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa39c0a2880 0x7fa39c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.113+0000 7fa3a37fe640 1 -- 172.21.15.78:0/3206564613 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa3a80f0b50 con 0x7fa3b41022e0 2024-03-16T11:03:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.113+0000 7fa3b1d9b640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa39c0a2880 0x7fa39c0a4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa3b41990a0 tx=0x7fa3a4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:35.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.117+0000 7fa3a37fe640 1 -- 172.21.15.78:0/3206564613 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa3a804d020 con 0x7fa3b41022e0 2024-03-16T11:03:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:35 smithi078 ceph-mon[32235]: Deploying daemon mds.e on smithi098 2024-03-16T11:03:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:35 smithi078 ceph-mon[32235]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:35.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.330+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3b40623e0 con 0x7fa3b41022e0 2024-03-16T11:03:35.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.330+0000 7fa3a37fe640 1 -- 172.21.15.78:0/3206564613 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7fa3a80be090 con 0x7fa3b41022e0 2024-03-16T11:03:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:35.331 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:03:35.331 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 5 2024-03-16T11:03:35.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa39c0a2880 msgr2=0x7fa39c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:35.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa39c0a2880 0x7fa39c0a4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa3b41990a0 tx=0x7fa3a4008040 comp rx=0 tx=0).stop 2024-03-16T11:03:35.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 msgr2=0x7fa3b4197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:35.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b4197b90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8036040 tx=0x7fa3a80042f0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 shutdown_connections 2024-03-16T11:03:35.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa39c0a2880 0x7fa39c0a4d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4107540 0x7fa3b419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b4104c10 0x7fa3b41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 --2- 172.21.15.78:0/3206564613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b41022e0 0x7fa3b4197b90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:35.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.332+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 >> 172.21.15.78:0/3206564613 conn(0x7fa3b40fbc70 msgr2=0x7fa3b4102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:35.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.333+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 shutdown_connections 2024-03-16T11:03:35.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:35.333+0000 7fa3b8ba9640 1 -- 172.21.15.78:0/3206564613 wait complete. 2024-03-16T11:03:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:35 smithi098 ceph-mon[35848]: Deploying daemon mds.e on smithi098 2024-03-16T11:03:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:35 smithi098 ceph-mon[35848]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:35.465 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 systemd[1]: Starting Ceph mds.e for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:35 smithi153 ceph-mon[35521]: Deploying daemon mds.e on smithi098 2024-03-16T11:03:35.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:35 smithi153 ceph-mon[35521]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:35.735 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 podman[66370]: 2024-03-16 11:03:35.465342488 +0000 UTC m=+0.113144686 container create 84f36ef85f500b1579fc9dd8f56e701585ff618a7763262cb7cb7f16d68a564f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:03:35.735 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 podman[66370]: 2024-03-16 11:03:35.371431307 +0000 UTC m=+0.019233500 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:35.735 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 podman[66370]: 2024-03-16 11:03:35.734873815 +0000 UTC m=+0.382676007 container init 84f36ef85f500b1579fc9dd8f56e701585ff618a7763262cb7cb7f16d68a564f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:03:36.184 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 podman[66370]: 2024-03-16 11:03:35.737403725 +0000 UTC m=+0.385205919 container start 84f36ef85f500b1579fc9dd8f56e701585ff618a7763262cb7cb7f16d68a564f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T11:03:36.184 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 ceph-mds[66389]: -- 172.21.15.98:0/917197188 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7ad1e3a40 con 0x55b7ad1b6c00 2024-03-16T11:03:36.184 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e[66385]: starting mds.e at 2024-03-16T11:03:36.184 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 bash[66370]: 84f36ef85f500b1579fc9dd8f56e701585ff618a7763262cb7cb7f16d68a564f 2024-03-16T11:03:36.184 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:03:35 smithi098 systemd[1]: Started Ceph mds.e for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:36 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3206564613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:36.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:36.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:36 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:36.516 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:36 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3206564613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:36.516 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.516 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.516 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.516 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:36.516 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:36.516 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:36 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:36.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:36 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3206564613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:36.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:36.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:36.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:36.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:36 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:37.344 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 -- 172.21.15.78:0/1008153410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b10101800 msgr2=0x7f8b10101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 --2- 172.21.15.78:0/1008153410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b10101800 0x7f8b10101c20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009900 tx=0x7f8b0402bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 -- 172.21.15.78:0/1008153410 shutdown_connections 2024-03-16T11:03:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 --2- 172.21.15.78:0/1008153410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b101033b0 0x7f8b10109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 --2- 172.21.15.78:0/1008153410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b101029f0 0x7f8b10102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 --2- 172.21.15.78:0/1008153410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b10101800 0x7f8b10101c20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 -- 172.21.15.78:0/1008153410 >> 172.21.15.78:0/1008153410 conn(0x7f8b100fcd90 msgr2=0x7f8b100ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:37.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 -- 172.21.15.78:0/1008153410 shutdown_connections 2024-03-16T11:03:37.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.457+0000 7f8b14929640 1 -- 172.21.15.78:0/1008153410 wait complete. 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 Processor -- start 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 -- start start 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b10101800 0x7f8b10199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101029f0 0x7f8b1019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b101033b0 0x7f8b101a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1010d300 con 0x7f8b101029f0 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1010d180 con 0x7f8b10101800 2024-03-16T11:03:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b14929640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1010d480 con 0x7f8b101033b0 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b0dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101029f0 0x7f8b1019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b0dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101029f0 0x7f8b1019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40160/0 (socket says 172.21.15.78:40160) 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b0dd9b640 1 -- 172.21.15.78:0/3596211889 learned_addr learned my addr 172.21.15.78:0/3596211889 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.458+0000 7f8b0e59c640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b10101800 0x7f8b10199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0ed9d640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b101033b0 0x7f8b101a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0dd9b640 1 -- 172.21.15.78:0/3596211889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b101033b0 msgr2=0x7f8b101a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0dd9b640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b101033b0 0x7f8b101a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0dd9b640 1 -- 172.21.15.78:0/3596211889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b10101800 msgr2=0x7f8b10199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0dd9b640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b10101800 0x7f8b10199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0dd9b640 1 -- 172.21.15.78:0/3596211889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b101a1890 con 0x7f8b101029f0 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0e59c640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b10101800 0x7f8b10199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b0dd9b640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101029f0 0x7f8b1019a1c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000e9f0 tx=0x7f8b0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8aff7fe640 1 -- 172.21.15.78:0/3596211889 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000cd80 con 0x7f8b101029f0 2024-03-16T11:03:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8aff7fe640 1 -- 172.21.15.78:0/3596211889 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b000108e0 con 0x7f8b101029f0 2024-03-16T11:03:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b101a1b80 con 0x7f8b101029f0 2024-03-16T11:03:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8aff7fe640 1 -- 172.21.15.78:0/3596211889 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00018920 con 0x7f8b101029f0 2024-03-16T11:03:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.459+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b101a20d0 con 0x7f8b101029f0 2024-03-16T11:03:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.460+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b101a22a0 con 0x7f8b101029f0 2024-03-16T11:03:37.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:37 smithi078 ceph-mon[32235]: Deploying daemon mds.f on smithi153 2024-03-16T11:03:37.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:37 smithi078 ceph-mon[32235]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:37.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:37 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6834/1190197046,v1:172.21.15.98:6835/1190197046] up:boot 2024-03-16T11:03:37.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:37 smithi078 ceph-mon[32235]: fsmap 5 up:standby 2024-03-16T11:03:37.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:37 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:03:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:37 smithi098 ceph-mon[35848]: Deploying daemon mds.f on smithi153 2024-03-16T11:03:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:37 smithi098 ceph-mon[35848]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:37.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:37 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6834/1190197046,v1:172.21.15.98:6835/1190197046] up:boot 2024-03-16T11:03:37.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:37 smithi098 ceph-mon[35848]: fsmap 5 up:standby 2024-03-16T11:03:37.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:37 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:03:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.461+0000 7f8aff7fe640 1 -- 172.21.15.78:0/3596211889 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8b00010a90 con 0x7f8b101029f0 2024-03-16T11:03:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.462+0000 7f8aff7fe640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8af80a2800 0x7f8af80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.462+0000 7f8b0e59c640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8af80a2800 0x7f8af80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.462+0000 7f8aff7fe640 1 -- 172.21.15.78:0/3596211889 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f8b000ceaa0 con 0x7f8b101029f0 2024-03-16T11:03:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.462+0000 7f8b0e59c640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8af80a2800 0x7f8af80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009230 tx=0x7f8b0402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:37.465 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:37 smithi153 ceph-mon[35521]: Deploying daemon mds.f on smithi153 2024-03-16T11:03:37.466 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:37 smithi153 ceph-mon[35521]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:37.466 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:37 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6834/1190197046,v1:172.21.15.98:6835/1190197046] up:boot 2024-03-16T11:03:37.466 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:37 smithi153 ceph-mon[35521]: fsmap 5 up:standby 2024-03-16T11:03:37.466 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:37 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:03:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.466+0000 7f8aff7fe640 1 -- 172.21.15.78:0/3596211889 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b0009bfe0 con 0x7f8b101029f0 2024-03-16T11:03:37.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.678+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b101a2470 con 0x7f8b101029f0 2024-03-16T11:03:37.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.678+0000 7f8aff7fe640 1 -- 172.21.15.78:0/3596211889 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f8b0009b8e0 con 0x7f8b101029f0 2024-03-16T11:03:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:37.679 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T11:03:37.679 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 6 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8af80a2800 msgr2=0x7f8af80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8af80a2800 0x7f8af80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009230 tx=0x7f8b0402c7e0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101029f0 msgr2=0x7f8b1019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101029f0 0x7f8b1019a1c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000e9f0 tx=0x7f8b0000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 shutdown_connections 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8af80a2800 0x7f8af80a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b101033b0 0x7f8b101a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101029f0 0x7f8b1019a1c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 --2- 172.21.15.78:0/3596211889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b10101800 0x7f8b10199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 >> 172.21.15.78:0/3596211889 conn(0x7f8b100fcd90 msgr2=0x7f8b10073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:37.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 shutdown_connections 2024-03-16T11:03:37.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:37.680+0000 7f8b14929640 1 -- 172.21.15.78:0/3596211889 wait complete. 2024-03-16T11:03:38.043 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:37 smithi153 systemd[1]: Starting Ceph mds.f for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:38.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:38 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3596211889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:38.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:38 smithi078 ceph-mon[32235]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:38.395 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:38 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3596211889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:38.395 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:38 smithi153 ceph-mon[35521]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:38.395 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 podman[66465]: 2024-03-16 11:03:38.046601344 +0000 UTC m=+0.019119465 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:38.395 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 podman[66465]: 2024-03-16 11:03:38.151384801 +0000 UTC m=+0.123902910 container create 32a197345a73294f2ddfe22bf72922244e66154febcd53aabb3b34f54aab2285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T11:03:38.395 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 podman[66465]: 2024-03-16 11:03:38.395409285 +0000 UTC m=+0.367927395 container init 32a197345a73294f2ddfe22bf72922244e66154febcd53aabb3b34f54aab2285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:03:38.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:38 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3596211889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:38.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:38 smithi098 ceph-mon[35848]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:38.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 podman[66465]: 2024-03-16 11:03:38.398045619 +0000 UTC m=+0.370563725 container start 32a197345a73294f2ddfe22bf72922244e66154febcd53aabb3b34f54aab2285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T11:03:38.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 ceph-mds[66485]: -- 172.21.15.153:0/4158412750 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565208e27a40 con 0x565208dfa800 2024-03-16T11:03:38.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f[66481]: starting mds.f at 2024-03-16T11:03:38.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 bash[66465]: 32a197345a73294f2ddfe22bf72922244e66154febcd53aabb3b34f54aab2285 2024-03-16T11:03:38.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:03:38 smithi153 systemd[1]: Started Ceph mds.f for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:39.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:39.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: Deploying daemon mds.c on smithi153 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6834/1109965992,v1:172.21.15.153:6835/1109965992] up:boot 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: fsmap 6 up:standby 2024-03-16T11:03:39.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:39 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:03:39.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: Deploying daemon mds.c on smithi153 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6834/1109965992,v1:172.21.15.153:6835/1109965992] up:boot 2024-03-16T11:03:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: fsmap 6 up:standby 2024-03-16T11:03:39.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:39 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: Deploying daemon mds.c on smithi153 2024-03-16T11:03:39.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6834/1109965992,v1:172.21.15.153:6835/1109965992] up:boot 2024-03-16T11:03:39.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: fsmap 6 up:standby 2024-03-16T11:03:39.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:39 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:03:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 -- 172.21.15.78:0/2481077257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 msgr2=0x7fb1ec109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/2481077257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec109c40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009ca0 tx=0x7fb1e402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 -- 172.21.15.78:0/2481077257 shutdown_connections 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/2481077257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec109c40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/2481077257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec1029f0 0x7fb1ec102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/2481077257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1ec101800 0x7fb1ec101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 -- 172.21.15.78:0/2481077257 >> 172.21.15.78:0/2481077257 conn(0x7fb1ec0fcd90 msgr2=0x7fb1ec0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 -- 172.21.15.78:0/2481077257 shutdown_connections 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.814+0000 7fb1f1d53640 1 -- 172.21.15.78:0/2481077257 wait complete. 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 Processor -- start 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 -- start start 2024-03-16T11:03:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1ec101800 0x7fb1ec1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec1029f0 0x7fb1ec197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec10d330 con 0x7fb1ec1033b0 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec10d1b0 con 0x7fb1ec101800 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1f1d53640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec10d4b0 con 0x7fb1ec1029f0 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1ebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1ebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40176/0 (socket says 172.21.15.78:40176) 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1ebfff640 1 -- 172.21.15.78:0/1883379678 learned_addr learned my addr 172.21.15.78:0/1883379678 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.815+0000 7fb1eaffd640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec1029f0 0x7fb1ec197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1eb7fe640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1ec101800 0x7fb1ec1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1ebfff640 1 -- 172.21.15.78:0/1883379678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec1029f0 msgr2=0x7fb1ec197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1ebfff640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec1029f0 0x7fb1ec197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1ebfff640 1 -- 172.21.15.78:0/1883379678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1ec101800 msgr2=0x7fb1ec1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1ebfff640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1ec101800 0x7fb1ec1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1ebfff640 1 -- 172.21.15.78:0/1883379678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec19f610 con 0x7fb1ec1033b0 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1eaffd640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec1029f0 0x7fb1ec197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1ebfff640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec19efa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb1e402c700 tx=0x7fb1e40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1e8ff9640 1 -- 172.21.15.78:0/1883379678 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4041020 con 0x7fb1ec1033b0 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1e8ff9640 1 -- 172.21.15.78:0/1883379678 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1e4003e20 con 0x7fb1ec1033b0 2024-03-16T11:03:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1e8ff9640 1 -- 172.21.15.78:0/1883379678 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4039040 con 0x7fb1ec1033b0 2024-03-16T11:03:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec19f8a0 con 0x7fb1ec1033b0 2024-03-16T11:03:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.816+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec19fd90 con 0x7fb1ec1033b0 2024-03-16T11:03:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.817+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ec108b20 con 0x7fb1ec1033b0 2024-03-16T11:03:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.819+0000 7fb1e8ff9640 1 -- 172.21.15.78:0/1883379678 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb1e4044020 con 0x7fb1ec1033b0 2024-03-16T11:03:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.820+0000 7fb1e8ff9640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb1d40a2800 0x7fb1d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.820+0000 7fb1eb7fe640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb1d40a2800 0x7fb1d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.820+0000 7fb1e8ff9640 1 -- 172.21.15.78:0/1883379678 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fb1e40f0b50 con 0x7fb1ec1033b0 2024-03-16T11:03:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.820+0000 7fb1eb7fe640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb1d40a2800 0x7fb1d40a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec102850 tx=0x7fb1dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:39.823+0000 7fb1e8ff9640 1 -- 172.21.15.78:0/1883379678 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1e404b020 con 0x7fb1ec1033b0 2024-03-16T11:03:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.036+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1ec02d370 con 0x7fb1ec1033b0 2024-03-16T11:03:40.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.036+0000 7fb1e8ff9640 1 -- 172.21.15.78:0/1883379678 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7fb1ec02d370 con 0x7fb1ec1033b0 2024-03-16T11:03:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:40.037 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-16T11:03:40.038 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 7 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb1d40a2800 msgr2=0x7fb1d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb1d40a2800 0x7fb1d40a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec102850 tx=0x7fb1dc005f90 comp rx=0 tx=0).stop 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 msgr2=0x7fb1ec19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec19efa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb1e402c700 tx=0x7fb1e40328a0 comp rx=0 tx=0).stop 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 shutdown_connections 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1ec1033b0 0x7fb1ec19efa0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb1d40a2800 0x7fb1d40a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:40.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec1029f0 0x7fb1ec197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:40.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 --2- 172.21.15.78:0/1883379678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1ec101800 0x7fb1ec1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:40.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 >> 172.21.15.78:0/1883379678 conn(0x7fb1ec0fcd90 msgr2=0x7fb1ec105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:40.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 shutdown_connections 2024-03-16T11:03:40.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:40.038+0000 7fb1f1d53640 1 -- 172.21.15.78:0/1883379678 wait complete. 2024-03-16T11:03:40.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:40 smithi153 ceph-mon[35521]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:40.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:40 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1883379678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:40.794 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:40 smithi153 systemd[1]: Starting Ceph mds.c for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:40.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:40 smithi078 ceph-mon[32235]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:40.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:40 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1883379678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:40.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:40 smithi098 ceph-mon[35848]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:40.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:40 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1883379678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:41.149 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:40 smithi153 podman[66773]: 2024-03-16 11:03:40.846426625 +0000 UTC m=+0.109679527 container create 43af1a72d8d1fbacc50525a72ab2ca50f1bdb026f32afa00095fd3904a1eae13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-16T11:03:41.150 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:40 smithi153 podman[66773]: 2024-03-16 11:03:40.75457487 +0000 UTC m=+0.017827780 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:41.150 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:41 smithi153 podman[66773]: 2024-03-16 11:03:41.057360676 +0000 UTC m=+0.320613578 container init 43af1a72d8d1fbacc50525a72ab2ca50f1bdb026f32afa00095fd3904a1eae13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T11:03:41.150 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:41 smithi153 podman[66773]: 2024-03-16 11:03:41.059966548 +0000 UTC m=+0.323219459 container start 43af1a72d8d1fbacc50525a72ab2ca50f1bdb026f32afa00095fd3904a1eae13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T11:03:41.150 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:41 smithi153 ceph-mds[66793]: -- 172.21.15.153:0/4182305696 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55661f965a40 con 0x55661f938c00 2024-03-16T11:03:41.150 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c[66789]: starting mds.c at 2024-03-16T11:03:41.150 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:41 smithi153 bash[66773]: 43af1a72d8d1fbacc50525a72ab2ca50f1bdb026f32afa00095fd3904a1eae13 2024-03-16T11:03:41.543 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:03:41 smithi153 systemd[1]: Started Ceph mds.c for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:42.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 -- 172.21.15.78:0/3654312993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8100780 msgr2=0x7fa6e8100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 --2- 172.21.15.78:0/3654312993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8100780 0x7fa6e8100ba0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009900 tx=0x7fa6dc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 -- 172.21.15.78:0/3654312993 shutdown_connections 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 --2- 172.21.15.78:0/3654312993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6e8102330 0x7fa6e8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 --2- 172.21.15.78:0/3654312993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8101970 0x7fa6e8101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 --2- 172.21.15.78:0/3654312993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8100780 0x7fa6e8100ba0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 -- 172.21.15.78:0/3654312993 >> 172.21.15.78:0/3654312993 conn(0x7fa6e80fbd10 msgr2=0x7fa6e80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 -- 172.21.15.78:0/3654312993 shutdown_connections 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.168+0000 7fa6eef77640 1 -- 172.21.15.78:0/3654312993 wait complete. 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 Processor -- start 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 -- start start 2024-03-16T11:03:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8100780 0x7fa6e8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8101970 0x7fa6e8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6e8102330 0x7fa6e819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e810c210 con 0x7fa6e8101970 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e810c090 con 0x7fa6e8100780 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6eef77640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e810c390 con 0x7fa6e8102330 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8101970 0x7fa6e8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8101970 0x7fa6e8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40196/0 (socket says 172.21.15.78:40196) 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6e7fff640 1 -- 172.21.15.78:0/2026428438 learned_addr learned my addr 172.21.15.78:0/2026428438 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6ed514640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6e8102330 0x7fa6e819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.169+0000 7fa6ecd13640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8100780 0x7fa6e8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e7fff640 1 -- 172.21.15.78:0/2026428438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6e8102330 msgr2=0x7fa6e819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e7fff640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6e8102330 0x7fa6e819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e7fff640 1 -- 172.21.15.78:0/2026428438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8100780 msgr2=0x7fa6e8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e7fff640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8100780 0x7fa6e8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e7fff640 1 -- 172.21.15.78:0/2026428438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e819f690 con 0x7fa6e8101970 2024-03-16T11:03:42.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6ecd13640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8100780 0x7fa6e8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:42.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e7fff640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8101970 0x7fa6e8198000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400d950 tx=0x7fa6d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:42.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e5ffb640 1 -- 172.21.15.78:0/2026428438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4017070 con 0x7fa6e8101970 2024-03-16T11:03:42.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e819f980 con 0x7fa6e8101970 2024-03-16T11:03:42.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e5ffb640 1 -- 172.21.15.78:0/2026428438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6d4009c30 con 0x7fa6e8101970 2024-03-16T11:03:42.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6e5ffb640 1 -- 172.21.15.78:0/2026428438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d400f620 con 0x7fa6e8101970 2024-03-16T11:03:42.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.170+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e819ff20 con 0x7fa6e8101970 2024-03-16T11:03:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.171+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e81a0150 con 0x7fa6e8101970 2024-03-16T11:03:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.172+0000 7fa6e5ffb640 1 -- 172.21.15.78:0/2026428438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa6d40109a0 con 0x7fa6e8101970 2024-03-16T11:03:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.173+0000 7fa6e5ffb640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6d80a2800 0x7fa6d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.173+0000 7fa6ecd13640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6d80a2800 0x7fa6d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.173+0000 7fa6e5ffb640 1 -- 172.21.15.78:0/2026428438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa6d40ce210 con 0x7fa6e8101970 2024-03-16T11:03:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.173+0000 7fa6ecd13640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6d80a2800 0x7fa6d80a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc006010 tx=0x7fa6dc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.177+0000 7fa6e5ffb640 1 -- 172.21.15.78:0/2026428438 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa6d4009de0 con 0x7fa6e8101970 2024-03-16T11:03:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.391+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6e81a0320 con 0x7fa6e8101970 2024-03-16T11:03:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.391+0000 7fa6e5ffb640 1 -- 172.21.15.78:0/2026428438 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7fa6d409b750 con 0x7fa6e8101970 2024-03-16T11:03:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:42.392 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-16T11:03:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-03-16T11:03:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6d80a2800 msgr2=0x7fa6d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6d80a2800 0x7fa6d80a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc006010 tx=0x7fa6dc02c7e0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8101970 msgr2=0x7fa6e8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8101970 0x7fa6e8198000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400d950 tx=0x7fa6d400de10 comp rx=0 tx=0).stop 2024-03-16T11:03:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 shutdown_connections 2024-03-16T11:03:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa6d80a2800 0x7fa6d80a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6e8102330 0x7fa6e819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6e8101970 0x7fa6e8198000 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 --2- 172.21.15.78:0/2026428438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e8100780 0x7fa6e8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 >> 172.21.15.78:0/2026428438 conn(0x7fa6e80fbd10 msgr2=0x7fa6e8069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.393+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 shutdown_connections 2024-03-16T11:03:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:42.394+0000 7fa6eef77640 1 -- 172.21.15.78:0/2026428438 wait complete. 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: Deploying daemon mds.h on smithi098 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6836/3505861862,v1:172.21.15.153:6837/3505861862] up:boot 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: fsmap 7 up:standby 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:03:42.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:42 smithi078 ceph-mon[32235]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:42.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:42.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:42.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:42.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: Deploying daemon mds.h on smithi098 2024-03-16T11:03:42.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6836/3505861862,v1:172.21.15.153:6837/3505861862] up:boot 2024-03-16T11:03:42.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: fsmap 7 up:standby 2024-03-16T11:03:42.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:03:42.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:42 smithi098 ceph-mon[35848]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:42.467 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: Deploying daemon mds.h on smithi098 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6836/3505861862,v1:172.21.15.153:6837/3505861862] up:boot 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: fsmap 7 up:standby 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:03:42.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:42 smithi153 ceph-mon[35521]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:43.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:43 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2026428438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:43.465 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 systemd[1]: Starting Ceph mds.h for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:43 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2026428438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:43 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2026428438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:43.873 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 podman[66681]: 2024-03-16 11:03:43.521767119 +0000 UTC m=+0.017781561 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:43.874 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 podman[66681]: 2024-03-16 11:03:43.620961083 +0000 UTC m=+0.116975524 container create 3fcbdba0c316768b1112d792005fe04394c94d1290eec7870722ab7c24f70a2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T11:03:43.874 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 podman[66681]: 2024-03-16 11:03:43.848721573 +0000 UTC m=+0.344736013 container init 3fcbdba0c316768b1112d792005fe04394c94d1290eec7870722ab7c24f70a2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:03:43.874 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 podman[66681]: 2024-03-16 11:03:43.85119921 +0000 UTC m=+0.347213652 container start 3fcbdba0c316768b1112d792005fe04394c94d1290eec7870722ab7c24f70a2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:03:43.874 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 ceph-mds[66700]: -- 172.21.15.98:0/485115124 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b400b63a40 con 0x55b400b36800 2024-03-16T11:03:44.214 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h[66696]: starting mds.h at 2024-03-16T11:03:44.214 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 bash[66681]: 3fcbdba0c316768b1112d792005fe04394c94d1290eec7870722ab7c24f70a2d 2024-03-16T11:03:44.214 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:03:43 smithi098 systemd[1]: Started Ceph mds.h for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:44.406 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:44.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 -- 172.21.15.78:0/1218769197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100700 msgr2=0x7f2b40100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:44.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 --2- 172.21.15.78:0/1218769197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100700 0x7f2b40100b20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009900 tx=0x7f2b3c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:44.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 -- 172.21.15.78:0/1218769197 shutdown_connections 2024-03-16T11:03:44.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 --2- 172.21.15.78:0/1218769197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b401022b0 0x7f2b40108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 --2- 172.21.15.78:0/1218769197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b401018f0 0x7f2b40101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 --2- 172.21.15.78:0/1218769197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100700 0x7f2b40100b20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 -- 172.21.15.78:0/1218769197 >> 172.21.15.78:0/1218769197 conn(0x7f2b400fbc70 msgr2=0x7f2b400fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 -- 172.21.15.78:0/1218769197 shutdown_connections 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.522+0000 7f2b48590640 1 -- 172.21.15.78:0/1218769197 wait complete. 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 Processor -- start 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 -- start start 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b40100700 0x7f2b400694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b401018f0 0x7f2b4019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b401022b0 0x7f2b4019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010c230 con 0x7f2b401022b0 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010c0b0 con 0x7f2b401018f0 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b48590640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010c3b0 con 0x7f2b40100700 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b46b2d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b401022b0 0x7f2b4019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b46b2d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b401022b0 0x7f2b4019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40214/0 (socket says 172.21.15.78:40214) 2024-03-16T11:03:44.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b46b2d640 1 -- 172.21.15.78:0/3106307384 learned_addr learned my addr 172.21.15.78:0/3106307384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b45b2b640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b401018f0 0x7f2b4019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.523+0000 7f2b4632c640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b40100700 0x7f2b400694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b46b2d640 1 -- 172.21.15.78:0/3106307384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b40100700 msgr2=0x7f2b400694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b46b2d640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b40100700 0x7f2b400694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b46b2d640 1 -- 172.21.15.78:0/3106307384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b401018f0 msgr2=0x7f2b4019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b46b2d640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b401018f0 0x7f2b4019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b46b2d640 1 -- 172.21.15.78:0/3106307384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4019f6e0 con 0x7f2b401022b0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b46b2d640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b401022b0 0x7f2b4019edd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800eb10 tx=0x7f2b3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b377fe640 1 -- 172.21.15.78:0/3106307384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38004b20 con 0x7f2b401022b0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4019f970 con 0x7f2b401022b0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b377fe640 1 -- 172.21.15.78:0/3106307384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b38004cc0 con 0x7f2b401022b0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b377fe640 1 -- 172.21.15.78:0/3106307384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38010690 con 0x7f2b401022b0 2024-03-16T11:03:44.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.524+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4019fec0 con 0x7f2b401022b0 2024-03-16T11:03:44.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.525+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b40107af0 con 0x7f2b401022b0 2024-03-16T11:03:44.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.526+0000 7f2b377fe640 1 -- 172.21.15.78:0/3106307384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2b38005000 con 0x7f2b401022b0 2024-03-16T11:03:44.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.527+0000 7f2b377fe640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2b2c0a2800 0x7f2b2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:44.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.527+0000 7f2b4632c640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2b2c0a2800 0x7f2b2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:44.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.527+0000 7f2b377fe640 1 -- 172.21.15.78:0/3106307384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f2b380cf000 con 0x7f2b401022b0 2024-03-16T11:03:44.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.527+0000 7f2b4632c640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2b2c0a2800 0x7f2b2c0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c005ab0 tx=0x7f2b3c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:44.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.530+0000 7f2b377fe640 1 -- 172.21.15.78:0/3106307384 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b3809c540 con 0x7f2b401022b0 2024-03-16T11:03:44.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.740+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b4002d400 con 0x7f2b401022b0 2024-03-16T11:03:44.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.741+0000 7f2b377fe640 1 -- 172.21.15.78:0/3106307384 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7f2b3809be40 con 0x7f2b401022b0 2024-03-16T11:03:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:44.742 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T11:03:44.742 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 9 2024-03-16T11:03:44.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2b2c0a2800 msgr2=0x7f2b2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:44.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2b2c0a2800 0x7f2b2c0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c005ab0 tx=0x7f2b3c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b401022b0 msgr2=0x7f2b4019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b401022b0 0x7f2b4019edd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800eb10 tx=0x7f2b3800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 shutdown_connections 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b401022b0 0x7f2b4019edd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b401018f0 0x7f2b4019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2b2c0a2800 0x7f2b2c0a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 --2- 172.21.15.78:0/3106307384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b40100700 0x7f2b400694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 >> 172.21.15.78:0/3106307384 conn(0x7f2b400fbc70 msgr2=0x7f2b40104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 shutdown_connections 2024-03-16T11:03:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:44.743+0000 7f2b48590640 1 -- 172.21.15.78:0/3106307384 wait complete. 2024-03-16T11:03:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:45.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:45.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: Deploying daemon mds.i on smithi153 2024-03-16T11:03:45.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] up:boot 2024-03-16T11:03:45.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: fsmap 8 up:standby 2024-03-16T11:03:45.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:03:45.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3106307384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:45.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: Deploying daemon mds.i on smithi153 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] up:boot 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: fsmap 8 up:standby 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:03:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3106307384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: Deploying daemon mds.i on smithi153 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] up:boot 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: fsmap 8 up:standby 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:03:45.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3106307384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:46.274 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 systemd[1]: Starting Ceph mds.i for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:46.543 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 podman[67087]: 2024-03-16 11:03:46.274138915 +0000 UTC m=+0.109562389 container create 2c6276463a8100e42a223000648676126d27aa0a2e6d60c976872fdf0833b255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T11:03:46.544 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 podman[67087]: 2024-03-16 11:03:46.182708344 +0000 UTC m=+0.018131823 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:46.544 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 podman[67087]: 2024-03-16 11:03:46.49360953 +0000 UTC m=+0.329033002 container init 2c6276463a8100e42a223000648676126d27aa0a2e6d60c976872fdf0833b255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T11:03:46.544 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 podman[67087]: 2024-03-16 11:03:46.496176494 +0000 UTC m=+0.331599974 container start 2c6276463a8100e42a223000648676126d27aa0a2e6d60c976872fdf0833b255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:03:46.544 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 ceph-mds[67106]: -- 172.21.15.153:0/3537822769 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556c947ffa40 con 0x556c947d3000 2024-03-16T11:03:46.544 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i[67102]: starting mds.i at 2024-03-16T11:03:46.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 -- 172.21.15.78:0/2402442343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c20100700 msgr2=0x7f3c20100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 --2- 172.21.15.78:0/2402442343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c20100700 0x7f3c20100b20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009900 tx=0x7f3c1002bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 -- 172.21.15.78:0/2402442343 shutdown_connections 2024-03-16T11:03:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 --2- 172.21.15.78:0/2402442343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c201022b0 0x7f3c20108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 --2- 172.21.15.78:0/2402442343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c201018f0 0x7f3c20101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 --2- 172.21.15.78:0/2402442343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c20100700 0x7f3c20100b20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 -- 172.21.15.78:0/2402442343 >> 172.21.15.78:0/2402442343 conn(0x7f3c200fbc70 msgr2=0x7f3c200fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 -- 172.21.15.78:0/2402442343 shutdown_connections 2024-03-16T11:03:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.866+0000 7f3c2636f640 1 -- 172.21.15.78:0/2402442343 wait complete. 2024-03-16T11:03:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 Processor -- start 2024-03-16T11:03:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 -- start start 2024-03-16T11:03:46.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c20100700 0x7f3c201969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:46.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c201018f0 0x7f3c20196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:46.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c201022b0 0x7f3c2019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:46.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2010c280 con 0x7f3c201018f0 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2010c100 con 0x7f3c20100700 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c2636f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2010c400 con 0x7f3c201022b0 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c1f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c201018f0 0x7f3c20196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c1f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c201018f0 0x7f3c20196f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43892/0 (socket says 172.21.15.78:43892) 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c1f7fe640 1 -- 172.21.15.78:0/174917511 learned_addr learned my addr 172.21.15.78:0/174917511 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.867+0000 7f3c1ffff640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c20100700 0x7f3c201969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c2490c640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c201022b0 0x7f3c2019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1f7fe640 1 -- 172.21.15.78:0/174917511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c201022b0 msgr2=0x7f3c2019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1f7fe640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c201022b0 0x7f3c2019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1f7fe640 1 -- 172.21.15.78:0/174917511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c20100700 msgr2=0x7f3c201969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1f7fe640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c20100700 0x7f3c201969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1f7fe640 1 -- 172.21.15.78:0/174917511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c2019e5f0 con 0x7f3c201018f0 2024-03-16T11:03:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c2490c640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c201022b0 0x7f3c2019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1ffff640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c20100700 0x7f3c201969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1f7fe640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c201018f0 0x7f3c20196f10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400e9f0 tx=0x7f3c1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1d7fa640 1 -- 172.21.15.78:0/174917511 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1400cd80 con 0x7f3c201018f0 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c2019e8e0 con 0x7f3c201018f0 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1d7fa640 1 -- 172.21.15.78:0/174917511 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3c140108e0 con 0x7f3c201018f0 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c1d7fa640 1 -- 172.21.15.78:0/174917511 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14018920 con 0x7f3c201018f0 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.868+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c2019ef00 con 0x7f3c201018f0 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.869+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c20107af0 con 0x7f3c201018f0 2024-03-16T11:03:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.870+0000 7f3c1d7fa640 1 -- 172.21.15.78:0/174917511 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3c14010a90 con 0x7f3c201018f0 2024-03-16T11:03:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.871+0000 7f3c1d7fa640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3c080a2800 0x7f3c080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.871+0000 7f3c1ffff640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3c080a2800 0x7f3c080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.871+0000 7f3c1d7fa640 1 -- 172.21.15.78:0/174917511 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f3c140ce960 con 0x7f3c201018f0 2024-03-16T11:03:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.871+0000 7f3c1ffff640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3c080a2800 0x7f3c080a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c10005ab0 tx=0x7f3c10005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:46.875+0000 7f3c1d7fa640 1 -- 172.21.15.78:0/174917511 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3c1409bea0 con 0x7f3c201018f0 2024-03-16T11:03:47.000 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 bash[67087]: 2c6276463a8100e42a223000648676126d27aa0a2e6d60c976872fdf0833b255 2024-03-16T11:03:47.000 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:03:46 smithi153 systemd[1]: Started Ceph mds.i for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:47.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:47 smithi078 ceph-mon[32235]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:47.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:47.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:47.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:47 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:47.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.097+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c2002d3d0 con 0x7f3c201018f0 2024-03-16T11:03:47.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.097+0000 7f3c1d7fa640 1 -- 172.21.15.78:0/174917511 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f3c1409b7a0 con 0x7f3c201018f0 2024-03-16T11:03:47.098 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:47.098 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:03:47.098 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 10 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3c080a2800 msgr2=0x7f3c080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3c080a2800 0x7f3c080a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c10005ab0 tx=0x7f3c10005fd0 comp rx=0 tx=0).stop 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c201018f0 msgr2=0x7f3c20196f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c201018f0 0x7f3c20196f10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400e9f0 tx=0x7f3c1400eeb0 comp rx=0 tx=0).stop 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 shutdown_connections 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3c080a2800 0x7f3c080a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c201022b0 0x7f3c2019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c201018f0 0x7f3c20196f10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 --2- 172.21.15.78:0/174917511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c20100700 0x7f3c201969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 >> 172.21.15.78:0/174917511 conn(0x7f3c200fbc70 msgr2=0x7f3c20104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 shutdown_connections 2024-03-16T11:03:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:47.099+0000 7f3c2636f640 1 -- 172.21.15.78:0/174917511 wait complete. 2024-03-16T11:03:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:47 smithi153 ceph-mon[35521]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:47.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:47.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:47 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:47 smithi098 ceph-mon[35848]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:47.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:47 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:48 smithi153 ceph-mon[35521]: Deploying daemon mds.g on smithi078 2024-03-16T11:03:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:48 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] up:boot 2024-03-16T11:03:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:48 smithi153 ceph-mon[35521]: fsmap 9 up:standby 2024-03-16T11:03:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:48 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:03:48.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:48 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/174917511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:48.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:48 smithi078 ceph-mon[32235]: Deploying daemon mds.g on smithi078 2024-03-16T11:03:48.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:48 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] up:boot 2024-03-16T11:03:48.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:48 smithi078 ceph-mon[32235]: fsmap 9 up:standby 2024-03-16T11:03:48.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:48 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:03:48.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:48 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/174917511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:48.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:48 smithi098 ceph-mon[35848]: Deploying daemon mds.g on smithi078 2024-03-16T11:03:48.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:48 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] up:boot 2024-03-16T11:03:48.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:48 smithi098 ceph-mon[35848]: fsmap 9 up:standby 2024-03-16T11:03:48.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:48 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:03:48.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:48 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/174917511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:49.112 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 -- 172.21.15.78:0/280597629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba41018f0 msgr2=0x7f8ba4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 --2- 172.21.15.78:0/280597629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba41018f0 0x7f8ba4101d70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009900 tx=0x7f8b9802bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 -- 172.21.15.78:0/280597629 shutdown_connections 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 --2- 172.21.15.78:0/280597629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ba41022b0 0x7f8ba4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 --2- 172.21.15.78:0/280597629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba41018f0 0x7f8ba4101d70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 --2- 172.21.15.78:0/280597629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ba4100700 0x7f8ba4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 -- 172.21.15.78:0/280597629 >> 172.21.15.78:0/280597629 conn(0x7f8ba40fbc70 msgr2=0x7f8ba40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 -- 172.21.15.78:0/280597629 shutdown_connections 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.220+0000 7f8ba929a640 1 -- 172.21.15.78:0/280597629 wait complete. 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 Processor -- start 2024-03-16T11:03:49.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 -- start start 2024-03-16T11:03:49.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba4100700 0x7f8ba4196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:49.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ba41018f0 0x7f8ba4196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:49.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ba41022b0 0x7f8ba419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:49.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba410c230 con 0x7f8ba4100700 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba410c0b0 con 0x7f8ba41018f0 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba929a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba410c3b0 con 0x7f8ba41022b0 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba4100700 0x7f8ba4196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba4100700 0x7f8ba4196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43904/0 (socket says 172.21.15.78:43904) 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.221+0000 7f8ba2ffd640 1 -- 172.21.15.78:0/192588027 learned_addr learned my addr 172.21.15.78:0/192588027 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba27fc640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ba41018f0 0x7f8ba4196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba37fe640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ba41022b0 0x7f8ba419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba2ffd640 1 -- 172.21.15.78:0/192588027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ba41022b0 msgr2=0x7f8ba419deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba2ffd640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ba41022b0 0x7f8ba419deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba2ffd640 1 -- 172.21.15.78:0/192588027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ba41018f0 msgr2=0x7f8ba4196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba2ffd640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ba41018f0 0x7f8ba4196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba2ffd640 1 -- 172.21.15.78:0/192588027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba419e520 con 0x7f8ba4100700 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba37fe640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ba41022b0 0x7f8ba419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba2ffd640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba4100700 0x7f8ba4196900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400d950 tx=0x7f8b9400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8b8bfff640 1 -- 172.21.15.78:0/192588027 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94017070 con 0x7f8ba4100700 2024-03-16T11:03:49.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8b8bfff640 1 -- 172.21.15.78:0/192588027 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b94009c30 con 0x7f8ba4100700 2024-03-16T11:03:49.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba419e810 con 0x7f8ba4100700 2024-03-16T11:03:49.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8b8bfff640 1 -- 172.21.15.78:0/192588027 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9400f620 con 0x7f8ba4100700 2024-03-16T11:03:49.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.222+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba419ece0 con 0x7f8ba4100700 2024-03-16T11:03:49.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.223+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba4107aa0 con 0x7f8ba4100700 2024-03-16T11:03:49.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.224+0000 7f8b8bfff640 1 -- 172.21.15.78:0/192588027 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8b9401a070 con 0x7f8ba4100700 2024-03-16T11:03:49.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.225+0000 7f8b8bfff640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2800 0x7f8b8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:49.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.225+0000 7f8ba27fc640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2800 0x7f8b8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:49.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.225+0000 7f8b8bfff640 1 -- 172.21.15.78:0/192588027 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f8b940cdff0 con 0x7f8ba4100700 2024-03-16T11:03:49.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.225+0000 7f8ba27fc640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2800 0x7f8b8c0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009230 tx=0x7f8b98005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:49.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.230+0000 7f8b8bfff640 1 -- 172.21.15.78:0/192588027 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b94009de0 con 0x7f8ba4100700 2024-03-16T11:03:49.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:49 smithi153 ceph-mon[35521]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:49.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:49 smithi078 ceph-mon[32235]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.450+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ba402d1b0 con 0x7f8ba4100700 2024-03-16T11:03:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.450+0000 7f8b8bfff640 1 -- 172.21.15.78:0/192588027 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f8b9409b530 con 0x7f8ba4100700 2024-03-16T11:03:49.451 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:49.451 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:03:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 10 2024-03-16T11:03:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2800 msgr2=0x7f8b8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2800 0x7f8b8c0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009230 tx=0x7f8b98005fd0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba4100700 msgr2=0x7f8ba4196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba4100700 0x7f8ba4196900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400d950 tx=0x7f8b9400de10 comp rx=0 tx=0).stop 2024-03-16T11:03:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 shutdown_connections 2024-03-16T11:03:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2800 0x7f8b8c0a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ba41022b0 0x7f8ba419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ba41018f0 0x7f8ba4196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 --2- 172.21.15.78:0/192588027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba4100700 0x7f8ba4196900 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.452+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 >> 172.21.15.78:0/192588027 conn(0x7f8ba40fbc70 msgr2=0x7f8ba4104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.453+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 shutdown_connections 2024-03-16T11:03:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:49.453+0000 7f8ba929a640 1 -- 172.21.15.78:0/192588027 wait complete. 2024-03-16T11:03:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:49 smithi098 ceph-mon[35848]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:49.809 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 16 11:03:49 smithi078 systemd[1]: Started Ceph mds.g for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:50 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/192588027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:50.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:50.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:50 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:50 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/192588027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:50.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:50 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:50.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:50 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/192588027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:50.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:50.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:50.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:50.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:50 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:51.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:51 smithi078 ceph-mon[32235]: Deploying daemon mds.a on smithi078 2024-03-16T11:03:51.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:51 smithi078 ceph-mon[32235]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:51.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:51 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.78:6838/3781553715,v1:172.21.15.78:6839/3781553715] up:boot 2024-03-16T11:03:51.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:51 smithi078 ceph-mon[32235]: fsmap 10 up:standby 2024-03-16T11:03:51.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:51 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:03:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:51 smithi153 ceph-mon[35521]: Deploying daemon mds.a on smithi078 2024-03-16T11:03:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:51 smithi153 ceph-mon[35521]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:51 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.78:6838/3781553715,v1:172.21.15.78:6839/3781553715] up:boot 2024-03-16T11:03:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:51 smithi153 ceph-mon[35521]: fsmap 10 up:standby 2024-03-16T11:03:51.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:51 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:03:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:51 smithi098 ceph-mon[35848]: Deploying daemon mds.a on smithi078 2024-03-16T11:03:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:51 smithi098 ceph-mon[35848]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:51 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.78:6838/3781553715,v1:172.21.15.78:6839/3781553715] up:boot 2024-03-16T11:03:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:51 smithi098 ceph-mon[35848]: fsmap 10 up:standby 2024-03-16T11:03:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:51 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:03:51.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 -- 172.21.15.78:0/167737781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e540ff840 msgr2=0x7f4e540ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/167737781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e540ff840 0x7f4e540ffcc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009900 tx=0x7f4e4002bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 -- 172.21.15.78:0/167737781 shutdown_connections 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/167737781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e54100200 0x7f4e54109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/167737781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e540ff840 0x7f4e540ffcc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/167737781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54102dd0 0x7f4e541031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 -- 172.21.15.78:0/167737781 >> 172.21.15.78:0/167737781 conn(0x7f4e540fb3e0 msgr2=0x7f4e540fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 -- 172.21.15.78:0/167737781 shutdown_connections 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.711+0000 7f4e5ad37640 1 -- 172.21.15.78:0/167737781 wait complete. 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 Processor -- start 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 -- start start 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e540ff840 0x7f4e54197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e54100200 0x7f4e54197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54102dd0 0x7f4e5419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5410d310 con 0x7f4e54100200 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5410d190 con 0x7f4e540ff840 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e5ad37640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5410d490 con 0x7f4e54102dd0 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e53fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e54100200 0x7f4e54197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e53fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e54100200 0x7f4e54197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43942/0 (socket says 172.21.15.78:43942) 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.712+0000 7f4e53fff640 1 -- 172.21.15.78:0/1027163791 learned_addr learned my addr 172.21.15.78:0/1027163791 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e592d4640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54102dd0 0x7f4e5419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e58ad3640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e540ff840 0x7f4e54197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e53fff640 1 -- 172.21.15.78:0/1027163791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54102dd0 msgr2=0x7f4e5419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e53fff640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54102dd0 0x7f4e5419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e53fff640 1 -- 172.21.15.78:0/1027163791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e540ff840 msgr2=0x7f4e54197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e53fff640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e540ff840 0x7f4e54197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e53fff640 1 -- 172.21.15.78:0/1027163791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5419f550 con 0x7f4e54100200 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e53fff640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e54100200 0x7f4e54197e70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e40005ab0 tx=0x7f4e400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e51ffb640 1 -- 172.21.15.78:0/1027163791 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40043020 con 0x7f4e54100200 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5419f7e0 con 0x7f4e54100200 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e51ffb640 1 -- 172.21.15.78:0/1027163791 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e400054c0 con 0x7f4e54100200 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e51ffb640 1 -- 172.21.15.78:0/1027163791 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40003df0 con 0x7f4e54100200 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.713+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e5419fd30 con 0x7f4e54100200 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.714+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e54108c30 con 0x7f4e54100200 2024-03-16T11:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.715+0000 7f4e51ffb640 1 -- 172.21.15.78:0/1027163791 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4e40046020 con 0x7f4e54100200 2024-03-16T11:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.716+0000 7f4e51ffb640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4e440a2800 0x7f4e440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.716+0000 7f4e51ffb640 1 -- 172.21.15.78:0/1027163791 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f4e400f1210 con 0x7f4e54100200 2024-03-16T11:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.716+0000 7f4e58ad3640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4e440a2800 0x7f4e440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.716+0000 7f4e58ad3640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4e440a2800 0x7f4e440a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4e48005fd0 tx=0x7f4e48005ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:51.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.720+0000 7f4e51ffb640 1 -- 172.21.15.78:0/1027163791 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e4004d020 con 0x7f4e54100200 2024-03-16T11:03:51.809 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:51 smithi078 systemd[1]: Starting Ceph mds.a for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:51.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.951+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e5402d1b0 con 0x7f4e54100200 2024-03-16T11:03:51.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.952+0000 7f4e51ffb640 1 -- 172.21.15.78:0/1027163791 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f4e400be750 con 0x7f4e54100200 2024-03-16T11:03:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:51.953 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:03:51.953 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 11 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4e440a2800 msgr2=0x7f4e440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4e440a2800 0x7f4e440a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4e48005fd0 tx=0x7f4e48005ce0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e54100200 msgr2=0x7f4e54197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e54100200 0x7f4e54197e70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e40005ab0 tx=0x7f4e400042f0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 shutdown_connections 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4e440a2800 0x7f4e440a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54102dd0 0x7f4e5419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e54100200 0x7f4e54197e70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 --2- 172.21.15.78:0/1027163791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e540ff840 0x7f4e54197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:51.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 >> 172.21.15.78:0/1027163791 conn(0x7f4e540fb3e0 msgr2=0x7f4e54105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:51.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 shutdown_connections 2024-03-16T11:03:51.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:51.954+0000 7f4e5ad37640 1 -- 172.21.15.78:0/1027163791 wait complete. 2024-03-16T11:03:52.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:52 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1027163791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:52.139 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:51 smithi078 podman[74579]: 2024-03-16 11:03:51.781237745 +0000 UTC m=+0.018750139 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:52.140 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:51 smithi078 podman[74579]: 2024-03-16 11:03:51.903136075 +0000 UTC m=+0.140648478 container create 67e9c56b4cf471beee5310dd0a1c937cc0c446799299680c13fc72fe01915b22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-16T11:03:52.140 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:52 smithi078 podman[74579]: 2024-03-16 11:03:52.13912251 +0000 UTC m=+0.376634906 container init 67e9c56b4cf471beee5310dd0a1c937cc0c446799299680c13fc72fe01915b22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:03:52.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:52 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1027163791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:52.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:52 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1027163791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:52.559 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:52 smithi078 podman[74579]: 2024-03-16 11:03:52.14204908 +0000 UTC m=+0.379561482 container start 67e9c56b4cf471beee5310dd0a1c937cc0c446799299680c13fc72fe01915b22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T11:03:52.559 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:52 smithi078 ceph-mds[74600]: -- 172.21.15.78:0/3966056210 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d228b9fa40 con 0x55d228b72800 2024-03-16T11:03:52.559 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:52 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a[74596]: starting mds.a at 2024-03-16T11:03:52.559 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:52 smithi078 bash[74579]: 67e9c56b4cf471beee5310dd0a1c937cc0c446799299680c13fc72fe01915b22 2024-03-16T11:03:52.560 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:03:52 smithi078 systemd[1]: Started Ceph mds.a for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:53.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:53 smithi098 ceph-mon[35848]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:53.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:53 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:53 smithi153 ceph-mon[35521]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:53 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:53 smithi078 ceph-mon[32235]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:03:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:03:53.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:53 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:53.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:54.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 -- 172.21.15.78:0/523023628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041018f0 msgr2=0x7f5204101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:54.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 --2- 172.21.15.78:0/523023628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041018f0 0x7f5204101d70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009900 tx=0x7f51ec02bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 -- 172.21.15.78:0/523023628 shutdown_connections 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 --2- 172.21.15.78:0/523023628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52041022b0 0x7f5204108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 --2- 172.21.15.78:0/523023628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041018f0 0x7f5204101d70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 --2- 172.21.15.78:0/523023628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204100700 0x7f5204100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 -- 172.21.15.78:0/523023628 >> 172.21.15.78:0/523023628 conn(0x7f52040fbc70 msgr2=0x7f52040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 -- 172.21.15.78:0/523023628 shutdown_connections 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 -- 172.21.15.78:0/523023628 wait complete. 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.084+0000 7f5209ab2640 1 Processor -- start 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5209ab2640 1 -- start start 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5209ab2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204100700 0x7f52040694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5209ab2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52041018f0 0x7f520419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5209ab2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041022b0 0x7f520419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:54.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5209ab2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f520410c290 con 0x7f52041022b0 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5209ab2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f520410c110 con 0x7f5204100700 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5209ab2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f520410c410 con 0x7f52041018f0 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041022b0 0x7f520419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041022b0 0x7f520419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43988/0 (socket says 172.21.15.78:43988) 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 -- 172.21.15.78:0/1420169486 learned_addr learned my addr 172.21.15.78:0/1420169486 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f52037fe640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204100700 0x7f52040694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5202ffd640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52041018f0 0x7f520419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 -- 172.21.15.78:0/1420169486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52041018f0 msgr2=0x7f520419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52041018f0 0x7f520419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 -- 172.21.15.78:0/1420169486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204100700 msgr2=0x7f52040694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204100700 0x7f52040694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 -- 172.21.15.78:0/1420169486 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520419f740 con 0x7f52041022b0 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5202ffd640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52041018f0 0x7f520419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f52037fe640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204100700 0x7f52040694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5203fff640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041022b0 0x7f520419ee30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009fd0 tx=0x7f51f400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.085+0000 7f5200ff9640 1 -- 172.21.15.78:0/1420169486 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4011830 con 0x7f52041022b0 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.086+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520419f9d0 con 0x7f52041022b0 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.086+0000 7f5200ff9640 1 -- 172.21.15.78:0/1420169486 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51f4016070 con 0x7f52041022b0 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.086+0000 7f5200ff9640 1 -- 172.21.15.78:0/1420169486 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4010de0 con 0x7f52041022b0 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.086+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520419ffa0 con 0x7f52041022b0 2024-03-16T11:03:54.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.087+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5204107aa0 con 0x7f52041022b0 2024-03-16T11:03:54.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.088+0000 7f5200ff9640 1 -- 172.21.15.78:0/1420169486 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f51f40119d0 con 0x7f52041022b0 2024-03-16T11:03:54.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.088+0000 7f5200ff9640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51f00a2800 0x7f51f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:54.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.088+0000 7f52037fe640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51f00a2800 0x7f51f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:54.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.089+0000 7f5200ff9640 1 -- 172.21.15.78:0/1420169486 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f51f40ce420 con 0x7f52041022b0 2024-03-16T11:03:54.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.089+0000 7f52037fe640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51f00a2800 0x7f51f00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5204101750 tx=0x7f51fc009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:54.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.092+0000 7f5200ff9640 1 -- 172.21.15.78:0/1420169486 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51f409b960 con 0x7f52041022b0 2024-03-16T11:03:54.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:54 smithi098 ceph-mon[35848]: Deploying daemon mds.b on smithi098 2024-03-16T11:03:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:54 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.78:6840/627084813,v1:172.21.15.78:6841/627084813] up:boot 2024-03-16T11:03:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:54 smithi098 ceph-mon[35848]: fsmap 11 up:standby 2024-03-16T11:03:54.215 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:54 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:03:54.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.307+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f520402d370 con 0x7f52041022b0 2024-03-16T11:03:54.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.308+0000 7f5200ff9640 1 -- 172.21.15.78:0/1420169486 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f51f409b260 con 0x7f52041022b0 2024-03-16T11:03:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:54.308 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:03:54.308 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 12 2024-03-16T11:03:54.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:54 smithi078 ceph-mon[32235]: Deploying daemon mds.b on smithi098 2024-03-16T11:03:54.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:54 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.78:6840/627084813,v1:172.21.15.78:6841/627084813] up:boot 2024-03-16T11:03:54.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:54 smithi078 ceph-mon[32235]: fsmap 11 up:standby 2024-03-16T11:03:54.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:54 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51f00a2800 msgr2=0x7f51f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51f00a2800 0x7f51f00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5204101750 tx=0x7f51fc009540 comp rx=0 tx=0).stop 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041022b0 msgr2=0x7f520419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041022b0 0x7f520419ee30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009fd0 tx=0x7f51f400b8e0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 shutdown_connections 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52041022b0 0x7f520419ee30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51f00a2800 0x7f51f00a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52041018f0 0x7f520419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 --2- 172.21.15.78:0/1420169486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204100700 0x7f52040694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 >> 172.21.15.78:0/1420169486 conn(0x7f52040fbc70 msgr2=0x7f5204104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 shutdown_connections 2024-03-16T11:03:54.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:54.310+0000 7f5209ab2640 1 -- 172.21.15.78:0/1420169486 wait complete. 2024-03-16T11:03:54.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:54 smithi153 ceph-mon[35521]: Deploying daemon mds.b on smithi098 2024-03-16T11:03:54.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:54 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.78:6840/627084813,v1:172.21.15.78:6841/627084813] up:boot 2024-03-16T11:03:54.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:54 smithi153 ceph-mon[35521]: fsmap 11 up:standby 2024-03-16T11:03:54.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:54 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:03:54.692 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:54 smithi098 systemd[1]: Starting Ceph mds.b for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:03:54.953 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:54 smithi098 podman[67003]: 2024-03-16 11:03:54.594672683 +0000 UTC m=+0.018310961 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:03:54.953 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:54 smithi098 podman[67003]: 2024-03-16 11:03:54.70057893 +0000 UTC m=+0.124217246 container create 90329a53074356e660e6c17e941564510f7084da9a27e960e4e48b82e71ebac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-16T11:03:54.953 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:54 smithi098 podman[67003]: 2024-03-16 11:03:54.92812192 +0000 UTC m=+0.351760197 container init 90329a53074356e660e6c17e941564510f7084da9a27e960e4e48b82e71ebac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-16T11:03:54.953 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:54 smithi098 podman[67003]: 2024-03-16 11:03:54.930525745 +0000 UTC m=+0.354164018 container start 90329a53074356e660e6c17e941564510f7084da9a27e960e4e48b82e71ebac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:03:55.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:55 smithi098 ceph-mon[35848]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:55.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:55 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1420169486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:55.215 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:54 smithi098 ceph-mds[67023]: -- 172.21.15.98:0/66612574 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa932a7a40 con 0x55fa9327a800 2024-03-16T11:03:55.215 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: starting mds.b at 2024-03-16T11:03:55.215 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:55 smithi098 bash[67003]: 90329a53074356e660e6c17e941564510f7084da9a27e960e4e48b82e71ebac1 2024-03-16T11:03:55.215 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:03:55 smithi098 systemd[1]: Started Ceph mds.b for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:03:55.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:55 smithi078 ceph-mon[32235]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:55.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:55 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1420169486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:55 smithi153 ceph-mon[35521]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:55 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1420169486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:56.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] up:boot 2024-03-16T11:03:56.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: fsmap 12 up:standby 2024-03-16T11:03:56.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:03:56.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:56 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:56.321 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:03:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.423+0000 7ff94a2b5640 1 -- 172.21.15.78:0/3953358152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 msgr2=0x7ff944108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.423+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/3953358152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff944108b60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff93c009ca0 tx=0x7ff93c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- 172.21.15.78:0/3953358152 shutdown_connections 2024-03-16T11:03:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/3953358152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff944108b60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/3953358152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff944101910 0x7ff944101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/3953358152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff944100720 0x7ff944100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- 172.21.15.78:0/3953358152 >> 172.21.15.78:0/3953358152 conn(0x7ff9440fbc90 msgr2=0x7ff9440fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- 172.21.15.78:0/3953358152 shutdown_connections 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- 172.21.15.78:0/3953358152 wait complete. 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 Processor -- start 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- start start 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff944100720 0x7ff944196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff944101910 0x7ff944196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff94419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff94410c230 con 0x7ff9441022d0 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff94410c0b0 con 0x7ff944100720 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.424+0000 7ff94a2b5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff94410c3b0 con 0x7ff944101910 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff94419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff94419def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35796/0 (socket says 172.21.15.78:35796) 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 -- 172.21.15.78:0/1370669928 learned_addr learned my addr 172.21.15.78:0/1370669928 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff9437fe640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff944101910 0x7ff944196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff943fff640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff944100720 0x7ff944196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 -- 172.21.15.78:0/1370669928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff944101910 msgr2=0x7ff944196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff944101910 0x7ff944196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 -- 172.21.15.78:0/1370669928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff944100720 msgr2=0x7ff944196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff944100720 0x7ff944196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 -- 172.21.15.78:0/1370669928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94419e560 con 0x7ff9441022d0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff948852640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff94419def0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff93c005cc0 tx=0x7ff93c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff9417fa640 1 -- 172.21.15.78:0/1370669928 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c041020 con 0x7ff9441022d0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94419e7f0 con 0x7ff9441022d0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff9417fa640 1 -- 172.21.15.78:0/1370669928 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff93c003e20 con 0x7ff9441022d0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff9417fa640 1 -- 172.21.15.78:0/1370669928 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c039040 con 0x7ff9441022d0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.425+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff94419ed00 con 0x7ff9441022d0 2024-03-16T11:03:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.426+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff944107b30 con 0x7ff9441022d0 2024-03-16T11:03:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.427+0000 7ff9417fa640 1 -- 172.21.15.78:0/1370669928 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff93c044020 con 0x7ff9441022d0 2024-03-16T11:03:56.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.428+0000 7ff9417fa640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff92c0a2800 0x7ff92c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.428+0000 7ff943fff640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff92c0a2800 0x7ff92c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.428+0000 7ff9417fa640 1 -- 172.21.15.78:0/1370669928 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7ff93c0f0bc0 con 0x7ff9441022d0 2024-03-16T11:03:56.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.428+0000 7ff943fff640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff92c0a2800 0x7ff92c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff944101770 tx=0x7ff934005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:56.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.431+0000 7ff9417fa640 1 -- 172.21.15.78:0/1370669928 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff93c04b020 con 0x7ff9441022d0 2024-03-16T11:03:56.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] up:boot 2024-03-16T11:03:56.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: fsmap 12 up:standby 2024-03-16T11:03:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:03:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:56 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:56.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] up:boot 2024-03-16T11:03:56.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: fsmap 12 up:standby 2024-03-16T11:03:56.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:03:56.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:56.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:56 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.659+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9440623e0 con 0x7ff9441022d0 2024-03-16T11:03:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.660+0000 7ff9417fa640 1 -- 172.21.15.78:0/1370669928 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7ff93c0be100 con 0x7ff9441022d0 2024-03-16T11:03:56.661 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:56.661 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-16T11:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 13 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff92c0a2800 msgr2=0x7ff92c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff92c0a2800 0x7ff92c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff944101770 tx=0x7ff934005f90 comp rx=0 tx=0).stop 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 msgr2=0x7ff94419def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff94419def0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff93c005cc0 tx=0x7ff93c0328a0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 shutdown_connections 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9441022d0 0x7ff94419def0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff92c0a2800 0x7ff92c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff944101910 0x7ff944196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 --2- 172.21.15.78:0/1370669928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff944100720 0x7ff944196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 >> 172.21.15.78:0/1370669928 conn(0x7ff9440fbc90 msgr2=0x7ff944104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 shutdown_connections 2024-03-16T11:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.662+0000 7ff94a2b5640 1 -- 172.21.15.78:0/1370669928 wait complete. 2024-03-16T11:03:56.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:03:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 -- 172.21.15.78:0/2827026150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e0100700 msgr2=0x7f15e0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 --2- 172.21.15.78:0/2827026150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e0100700 0x7f15e0100b20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009900 tx=0x7f15d802bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 -- 172.21.15.78:0/2827026150 shutdown_connections 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 --2- 172.21.15.78:0/2827026150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15e01022b0 0x7f15e0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 --2- 172.21.15.78:0/2827026150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e01018f0 0x7f15e0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 --2- 172.21.15.78:0/2827026150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e0100700 0x7f15e0100b20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 -- 172.21.15.78:0/2827026150 >> 172.21.15.78:0/2827026150 conn(0x7f15e00fbc70 msgr2=0x7f15e00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 -- 172.21.15.78:0/2827026150 shutdown_connections 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.784+0000 7f15e55f3640 1 -- 172.21.15.78:0/2827026150 wait complete. 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 Processor -- start 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 -- start start 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e0100700 0x7f15e0069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e01018f0 0x7f15e019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15e01022b0 0x7f15e019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f15e010c200 con 0x7f15e01018f0 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15e010c080 con 0x7f15e0100700 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15e55f3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15e010c380 con 0x7f15e01022b0 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e01018f0 0x7f15e019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e01018f0 0x7f15e019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35822/0 (socket says 172.21.15.78:35822) 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 -- 172.21.15.78:0/3415475046 learned_addr learned my addr 172.21.15.78:0/3415475046 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15deffd640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e0100700 0x7f15e0069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15df7fe640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15e01022b0 0x7f15e019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 -- 172.21.15.78:0/3415475046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15e01022b0 msgr2=0x7f15e019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15e01022b0 0x7f15e019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 -- 172.21.15.78:0/3415475046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e0100700 msgr2=0x7f15e0069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e0100700 0x7f15e0069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.785+0000 7f15de7fc640 1 -- 172.21.15.78:0/3415475046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e019f6b0 con 0x7f15e01018f0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.786+0000 7f15de7fc640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e01018f0 0x7f15e019c780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f15c800ebc0 tx=0x7f15c800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.786+0000 7f15d7fff640 1 -- 172.21.15.78:0/3415475046 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8010630 con 0x7f15e01018f0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.786+0000 7f15d7fff640 1 -- 172.21.15.78:0/3415475046 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15c8010c80 con 0x7f15e01018f0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.786+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e019f9a0 con 0x7f15e01018f0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.786+0000 7f15d7fff640 1 -- 172.21.15.78:0/3415475046 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8018b30 con 0x7f15e01018f0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.786+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e019fef0 con 0x7f15e01018f0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.787+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e0107b40 con 0x7f15e01018f0 2024-03-16T11:03:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.788+0000 7f15d7fff640 1 -- 172.21.15.78:0/3415475046 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f15c8018cd0 con 0x7f15e01018f0 2024-03-16T11:03:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.789+0000 7f15d7fff640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.789+0000 7f15d7fff640 1 -- 172.21.15.78:0/3415475046 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f15c80ceb50 con 0x7f15e01018f0 2024-03-16T11:03:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.789+0000 7f15deffd640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.789+0000 7f15deffd640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005ab0 tx=0x7f15d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:56.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:56.794+0000 7f15d7fff640 1 -- 172.21.15.78:0/3415475046 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f15c809ba40 con 0x7f15e01018f0 2024-03-16T11:03:57.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.009+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15e002d3b0 con 0x7f15e01018f0 2024-03-16T11:03:57.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.010+0000 7f15d7fff640 1 -- 172.21.15.78:0/3415475046 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18373 (secure 0 0 0) 0x7f15c809fae0 con 0x7f15e01018f0 2024-03-16T11:03:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:57.011 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:03:10.113262+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:03:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15cc0a2800 msgr2=0x7f15cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005ab0 tx=0x7f15d8005fd0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e01018f0 msgr2=0x7f15e019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e01018f0 0x7f15e019c780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f15c800ebc0 tx=0x7f15c800c6a0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 shutdown_connections 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15e01022b0 0x7f15e019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15e01018f0 0x7f15e019c780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 --2- 172.21.15.78:0/3415475046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e0100700 0x7f15e0069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 >> 172.21.15.78:0/3415475046 conn(0x7f15e00fbc70 msgr2=0x7f15e0104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 shutdown_connections 2024-03-16T11:03:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.012+0000 7f15e55f3640 1 -- 172.21.15.78:0/3415475046 wait complete. 2024-03-16T11:03:57.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:03:57.024 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:03:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 -- 172.21.15.78:0/339660460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4601006e0 msgr2=0x7fb460100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 --2- 172.21.15.78:0/339660460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4601006e0 0x7fb460100b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb458009900 tx=0x7fb45802bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 -- 172.21.15.78:0/339660460 shutdown_connections 2024-03-16T11:03:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 --2- 172.21.15.78:0/339660460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb460102290 0x7fb460108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 --2- 172.21.15.78:0/339660460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4601018d0 0x7fb460101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 --2- 172.21.15.78:0/339660460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4601006e0 0x7fb460100b00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 -- 172.21.15.78:0/339660460 >> 172.21.15.78:0/339660460 conn(0x7fb4600fbc70 msgr2=0x7fb4600fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:57.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 -- 172.21.15.78:0/339660460 shutdown_connections 2024-03-16T11:03:57.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.129+0000 7fb46510a640 1 -- 172.21.15.78:0/339660460 wait complete. 2024-03-16T11:03:57.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 Processor -- start 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 -- start start 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4601006e0 0x7fb4600694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4601018d0 0x7fb46019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb460102290 0x7fb46019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb46010c1e0 con 0x7fb460102290 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb46010c060 con 0x7fb4601006e0 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb46510a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb46010c360 con 0x7fb4601018d0 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb45f59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb460102290 0x7fb46019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb45f59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb460102290 0x7fb46019edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35836/0 (socket says 172.21.15.78:35836) 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb45f59e640 1 -- 172.21.15.78:0/55782917 learned_addr learned my addr 172.21.15.78:0/55782917 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb45e59c640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4601018d0 0x7fb46019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.130+0000 7fb45ed9d640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4601006e0 0x7fb4600694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb45f59e640 1 -- 172.21.15.78:0/55782917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4601018d0 msgr2=0x7fb46019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb45f59e640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4601018d0 0x7fb46019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb45f59e640 1 -- 172.21.15.78:0/55782917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4601006e0 msgr2=0x7fb4600694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb45f59e640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4601006e0 0x7fb4600694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb45f59e640 1 -- 172.21.15.78:0/55782917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46019f6b0 con 0x7fb460102290 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb45ed9d640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4601006e0 0x7fb4600694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb45f59e640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb460102290 0x7fb46019edf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb45000eaa0 tx=0x7fb45000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb457fff640 1 -- 172.21.15.78:0/55782917 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45000cdb0 con 0x7fb460102290 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46019f9a0 con 0x7fb460102290 2024-03-16T11:03:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb457fff640 1 -- 172.21.15.78:0/55782917 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb450005020 con 0x7fb460102290 2024-03-16T11:03:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb457fff640 1 -- 172.21.15.78:0/55782917 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4500116d0 con 0x7fb460102290 2024-03-16T11:03:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.131+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb46019fea0 con 0x7fb460102290 2024-03-16T11:03:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.132+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb460107b70 con 0x7fb460102290 2024-03-16T11:03:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.133+0000 7fb457fff640 1 -- 172.21.15.78:0/55782917 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb4500051d0 con 0x7fb460102290 2024-03-16T11:03:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.133+0000 7fb457fff640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb44c0a2800 0x7fb44c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.134+0000 7fb45ed9d640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb44c0a2800 0x7fb44c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.134+0000 7fb457fff640 1 -- 172.21.15.78:0/55782917 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fb4500ce7d0 con 0x7fb460102290 2024-03-16T11:03:57.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.134+0000 7fb45ed9d640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb44c0a2800 0x7fb44c0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb458006010 tx=0x7fb45802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:57.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.138+0000 7fb457fff640 1 -- 172.21.15.78:0/55782917 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb450016030 con 0x7fb460102290 2024-03-16T11:03:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1370669928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:57 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3415475046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.328+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb46002d3d0 con 0x7fb460102290 2024-03-16T11:03:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.329+0000 7fb457fff640 1 -- 172.21.15.78:0/55782917 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7fb45009bd10 con 0x7fb460102290 2024-03-16T11:03:57.329 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:03:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb44c0a2800 msgr2=0x7fb44c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb44c0a2800 0x7fb44c0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb458006010 tx=0x7fb45802c7e0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb460102290 msgr2=0x7fb46019edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb460102290 0x7fb46019edf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb45000eaa0 tx=0x7fb45000ef60 comp rx=0 tx=0).stop 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 shutdown_connections 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb460102290 0x7fb46019edf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb44c0a2800 0x7fb44c0a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4601018d0 0x7fb46019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 --2- 172.21.15.78:0/55782917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4601006e0 0x7fb4600694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 >> 172.21.15.78:0/55782917 conn(0x7fb4600fbc70 msgr2=0x7fb460104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 shutdown_connections 2024-03-16T11:03:57.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.331+0000 7fb46510a640 1 -- 172.21.15.78:0/55782917 wait complete. 2024-03-16T11:03:57.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T11:03:57.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.445+0000 7f1c54485640 1 -- 172.21.15.78:0/1977614331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c1022e0 msgr2=0x7f1c4c1046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.445+0000 7f1c54485640 1 --2- 172.21.15.78:0/1977614331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c1022e0 0x7f1c4c1046d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009900 tx=0x7f1c4802bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:57.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.445+0000 7f1c54485640 1 -- 172.21.15.78:0/1977614331 shutdown_connections 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.445+0000 7f1c54485640 1 --2- 172.21.15.78:0/1977614331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4c107540 0x7f1c4c109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.445+0000 7f1c54485640 1 --2- 172.21.15.78:0/1977614331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c104c10 0x7f1c4c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.445+0000 7f1c54485640 1 --2- 172.21.15.78:0/1977614331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c1022e0 0x7f1c4c1046d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.445+0000 7f1c54485640 1 -- 172.21.15.78:0/1977614331 >> 172.21.15.78:0/1977614331 conn(0x7f1c4c0fbc70 msgr2=0x7f1c4c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 -- 172.21.15.78:0/1977614331 shutdown_connections 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 -- 172.21.15.78:0/1977614331 wait complete. 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 Processor -- start 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 -- start start 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c1022e0 0x7f1c4c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c104c10 0x7f1c4c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4c107540 0x7f1c4c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c10d320 con 0x7f1c4c104c10 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c10d1a0 con 0x7f1c4c1022e0 2024-03-16T11:03:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c54485640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c10d4a0 con 0x7f1c4c107540 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c51a20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c104c10 0x7f1c4c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c51a20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c104c10 0x7f1c4c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35854/0 (socket says 172.21.15.78:35854) 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c51a20640 1 -- 172.21.15.78:0/518744966 learned_addr learned my addr 172.21.15.78:0/518744966 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c52221640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c1022e0 0x7f1c4c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.446+0000 7f1c52a22640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4c107540 0x7f1c4c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c51a20640 1 -- 172.21.15.78:0/518744966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4c107540 msgr2=0x7f1c4c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c51a20640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4c107540 0x7f1c4c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c51a20640 1 -- 172.21.15.78:0/518744966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c1022e0 msgr2=0x7f1c4c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c51a20640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c1022e0 0x7f1c4c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c51a20640 1 -- 172.21.15.78:0/518744966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4c19f6b0 con 0x7f1c4c104c10 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c52a22640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4c107540 0x7f1c4c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c51a20640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c104c10 0x7f1c4c197fd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00e9c0 tx=0x7f1c3c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c437fe640 1 -- 172.21.15.78:0/518744966 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c00cd80 con 0x7f1c4c104c10 2024-03-16T11:03:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c437fe640 1 -- 172.21.15.78:0/518744966 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c3c0054d0 con 0x7f1c4c104c10 2024-03-16T11:03:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4c19f9a0 con 0x7f1c4c104c10 2024-03-16T11:03:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c437fe640 1 -- 172.21.15.78:0/518744966 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c010430 con 0x7f1c4c104c10 2024-03-16T11:03:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.447+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4c19fef0 con 0x7f1c4c104c10 2024-03-16T11:03:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.448+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c4c0fd760 con 0x7f1c4c104c10 2024-03-16T11:03:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.449+0000 7f1c437fe640 1 -- 172.21.15.78:0/518744966 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1c3c004fb0 con 0x7f1c4c104c10 2024-03-16T11:03:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.449+0000 7f1c437fe640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c380a2800 0x7f1c380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.450+0000 7f1c52221640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c380a2800 0x7f1c380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.450+0000 7f1c437fe640 1 -- 172.21.15.78:0/518744966 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f1c3c0cecb0 con 0x7f1c4c104c10 2024-03-16T11:03:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.450+0000 7f1c52221640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c380a2800 0x7f1c380a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009230 tx=0x7f1c48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:57.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.453+0000 7f1c437fe640 1 -- 172.21.15.78:0/518744966 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c3c053740 con 0x7f1c4c104c10 2024-03-16T11:03:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1370669928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:57 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3415475046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1370669928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:03:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:57.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:57 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3415475046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:03:57.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.638+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1c4c02d400 con 0x7f1c4c104c10 2024-03-16T11:03:57.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.638+0000 7f1c437fe640 1 -- 172.21.15.78:0/518744966 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) ==== 84+0+65 (secure 0 0 0) 0x7f1c3c09c1d0 con 0x7f1c4c104c10 2024-03-16T11:03:57.639 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:03:57.639 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-03-16T11:03:57.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:03:57.639 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-03-16T11:03:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-03-16T11:03:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:03:57.640 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-03-16T11:03:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.640+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c380a2800 msgr2=0x7f1c380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.640+0000 7f1c54485640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c380a2800 0x7f1c380a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009230 tx=0x7f1c48005fd0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.640+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c104c10 msgr2=0x7f1c4c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.640+0000 7f1c54485640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c104c10 0x7f1c4c197fd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00e9c0 tx=0x7f1c3c00ee80 comp rx=0 tx=0).stop 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 shutdown_connections 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c380a2800 0x7f1c380a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4c107540 0x7f1c4c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c4c104c10 0x7f1c4c197fd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 --2- 172.21.15.78:0/518744966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c1022e0 0x7f1c4c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 >> 172.21.15.78:0/518744966 conn(0x7f1c4c0fbc70 msgr2=0x7f1c4c102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 shutdown_connections 2024-03-16T11:03:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.641+0000 7f1c54485640 1 -- 172.21.15.78:0/518744966 wait complete. 2024-03-16T11:03:57.653 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T11:03:57.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T11:03:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2862550604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be8102a00 msgr2=0x7f0be8102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2862550604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be8102a00 0x7f0be8102e80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009900 tx=0x7f0bd802bc30 comp rx=0 tx=0).stop 2024-03-16T11:03:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2862550604 shutdown_connections 2024-03-16T11:03:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2862550604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be81033c0 0x7f0be8109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2862550604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be8102a00 0x7f0be8102e80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2862550604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be8101810 0x7f0be8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2862550604 >> 172.21.15.78:0/2862550604 conn(0x7f0be80fcd80 msgr2=0x7f0be80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2862550604 shutdown_connections 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2862550604 wait complete. 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 Processor -- start 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.777+0000 7f0bef8ea640 1 -- start start 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be8101810 0x7f0be8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be8102a00 0x7f0be8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be81033c0 0x7f0be819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810d3a0 con 0x7f0be81033c0 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810d220 con 0x7f0be8102a00 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810d520 con 0x7f0be8101810 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be81033c0 0x7f0be819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be81033c0 0x7f0be819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35876/0 (socket says 172.21.15.78:35876) 2024-03-16T11:03:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 -- 172.21.15.78:0/2974644507 learned_addr learned my addr 172.21.15.78:0/2974644507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bece85640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be8102a00 0x7f0be8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bed686640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be8101810 0x7f0be8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 -- 172.21.15.78:0/2974644507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be8101810 msgr2=0x7f0be8197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be8101810 0x7f0be8197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 -- 172.21.15.78:0/2974644507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be8102a00 msgr2=0x7f0be8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be8102a00 0x7f0be8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 -- 172.21.15.78:0/2974644507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be819f6f0 con 0x7f0be81033c0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bede87640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be81033c0 0x7f0be819f080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0be000b970 tx=0x7f0be000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bde7fc640 1 -- 172.21.15.78:0/2974644507 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0019070 con 0x7f0be81033c0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bde7fc640 1 -- 172.21.15.78:0/2974644507 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0be0014440 con 0x7f0be81033c0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be819f9e0 con 0x7f0be81033c0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bde7fc640 1 -- 172.21.15.78:0/2974644507 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be00075d0 con 0x7f0be81033c0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.778+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be819ffd0 con 0x7f0be81033c0 2024-03-16T11:03:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.779+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be8108b50 con 0x7f0be81033c0 2024-03-16T11:03:57.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.780+0000 7f0bde7fc640 1 -- 172.21.15.78:0/2974644507 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0be001c070 con 0x7f0be81033c0 2024-03-16T11:03:57.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.781+0000 7f0bde7fc640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0bd40a2880 0x7f0bd40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.781+0000 7f0bed686640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0bd40a2880 0x7f0bd40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.781+0000 7f0bde7fc640 1 -- 172.21.15.78:0/2974644507 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f0be00cdf60 con 0x7f0be81033c0 2024-03-16T11:03:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.781+0000 7f0bed686640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0bd40a2880 0x7f0bd40a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0be8102860 tx=0x7f0be400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.784+0000 7f0bde7fc640 1 -- 172.21.15.78:0/2974644507 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0be009ae50 con 0x7f0be81033c0 2024-03-16T11:03:57.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:57.964+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0be80623e0 con 0x7f0be81033c0 2024-03-16T11:03:58.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.264+0000 7f0bde7fc640 1 -- 172.21.15.78:0/2974644507 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) ==== 138+0+0 (secure 0 0 0) 0x7f0be009eef0 con 0x7f0be81033c0 2024-03-16T11:03:58.264 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-03-16T11:03:58.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.265+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0bd40a2880 msgr2=0x7f0bd40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:58.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0bd40a2880 0x7f0bd40a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0be8102860 tx=0x7f0be400b390 comp rx=0 tx=0).stop 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be81033c0 msgr2=0x7f0be819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be81033c0 0x7f0be819f080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0be000b970 tx=0x7f0be000be30 comp rx=0 tx=0).stop 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 shutdown_connections 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be81033c0 0x7f0be819f080 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be8102a00 0x7f0be8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0bd40a2880 0x7f0bd40a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 --2- 172.21.15.78:0/2974644507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be8101810 0x7f0be8197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 >> 172.21.15.78:0/2974644507 conn(0x7f0be80fcd80 msgr2=0x7f0be8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 shutdown_connections 2024-03-16T11:03:58.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.266+0000 7f0bef8ea640 1 -- 172.21.15.78:0/2974644507 wait complete. 2024-03-16T11:03:58.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T11:03:58.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 -- 172.21.15.78:0/1564672713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9101022b0 msgr2=0x7fb910108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:58.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1564672713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9101022b0 0x7fb910108b40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb908009ca0 tx=0x7fb90802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 -- 172.21.15.78:0/1564672713 shutdown_connections 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1564672713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9101022b0 0x7fb910108b40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1564672713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9101018f0 0x7fb910101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1564672713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb910100700 0x7fb910100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 -- 172.21.15.78:0/1564672713 >> 172.21.15.78:0/1564672713 conn(0x7fb9100fbc90 msgr2=0x7fb9100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 -- 172.21.15.78:0/1564672713 shutdown_connections 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.401+0000 7fb916ba9640 1 -- 172.21.15.78:0/1564672713 wait complete. 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 Processor -- start 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 -- start start 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb910100700 0x7fb91006a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9101018f0 0x7fb91019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:58.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9101022b0 0x7fb91019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010c230 con 0x7fb910100700 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010c0b0 con 0x7fb9101022b0 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb916ba9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010c3b0 con 0x7fb9101018f0 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb914945640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb910100700 0x7fb91006a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb914945640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb910100700 0x7fb91006a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35908/0 (socket says 172.21.15.78:35908) 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb914945640 1 -- 172.21.15.78:0/1889975738 learned_addr learned my addr 172.21.15.78:0/1889975738 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb907fff640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9101018f0 0x7fb91019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb915146640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9101022b0 0x7fb91019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:58.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb914945640 1 -- 172.21.15.78:0/1889975738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9101018f0 msgr2=0x7fb91019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:58.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.402+0000 7fb914945640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9101018f0 0x7fb91019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.403+0000 7fb914945640 1 -- 172.21.15.78:0/1889975738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9101022b0 msgr2=0x7fb91019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:58.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.403+0000 7fb914945640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9101022b0 0x7fb91019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:58.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.403+0000 7fb914945640 1 -- 172.21.15.78:0/1889975738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91019f630 con 0x7fb910100700 2024-03-16T11:03:58.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.403+0000 7fb907fff640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9101018f0 0x7fb91019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:03:58.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.403+0000 7fb914945640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb910100700 0x7fb91006a6d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e9c0 tx=0x7fb90c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:58.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.403+0000 7fb905ffb640 1 -- 172.21.15.78:0/1889975738 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c00cd80 con 0x7fb910100700 2024-03-16T11:03:58.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.403+0000 7fb905ffb640 1 -- 172.21.15.78:0/1889975738 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb90c0035f0 con 0x7fb910100700 2024-03-16T11:03:58.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.404+0000 7fb905ffb640 1 -- 172.21.15.78:0/1889975738 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c010690 con 0x7fb910100700 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.404+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91019f920 con 0x7fb910100700 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.404+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb91019fe20 con 0x7fb910100700 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.405+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb910107a70 con 0x7fb910100700 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.406+0000 7fb905ffb640 1 -- 172.21.15.78:0/1889975738 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb90c0037a0 con 0x7fb910100700 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.407+0000 7fb905ffb640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8f80a2800 0x7fb8f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.407+0000 7fb905ffb640 1 -- 172.21.15.78:0/1889975738 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7497+0+0 (secure 0 0 0) 0x7fb90c0ceda0 con 0x7fb910100700 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.407+0000 7fb907fff640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8f80a2800 0x7fb8f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:58.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.407+0000 7fb907fff640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8f80a2800 0x7fb8f80a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb91006a440 tx=0x7fb90000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:58.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.413+0000 7fb905ffb640 1 -- 172.21.15.78:0/1889975738 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb90c09c170 con 0x7fb910100700 2024-03-16T11:03:58.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:58 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:58.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:58 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/55782917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:58.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:58 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/518744966' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T11:03:58.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:58 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:58.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:58 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/55782917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:58.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:58 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/518744966' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T11:03:58.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:58.605+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb91002d1b0 con 0x7fb910100700 2024-03-16T11:03:58.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:58 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:03:58.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:58 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/55782917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:03:58.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:58 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/518744966' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T11:03:59.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.267+0000 7fb905ffb640 1 -- 172.21.15.78:0/1889975738 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) ==== 170+0+0 (secure 0 0 0) 0x7fb90c09ba70 con 0x7fb910100700 2024-03-16T11:03:59.268 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8f80a2800 msgr2=0x7fb8f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8f80a2800 0x7fb8f80a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb91006a440 tx=0x7fb90000b390 comp rx=0 tx=0).stop 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb910100700 msgr2=0x7fb91006a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb910100700 0x7fb91006a6d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e9c0 tx=0x7fb90c00ee80 comp rx=0 tx=0).stop 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 shutdown_connections 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9101022b0 0x7fb91019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8f80a2800 0x7fb8f80a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9101018f0 0x7fb91019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 --2- 172.21.15.78:0/1889975738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb910100700 0x7fb91006a6d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 >> 172.21.15.78:0/1889975738 conn(0x7fb9100fbc90 msgr2=0x7fb910104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 shutdown_connections 2024-03-16T11:03:59.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.269+0000 7fb916ba9640 1 -- 172.21.15.78:0/1889975738 wait complete. 2024-03-16T11:03:59.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T11:03:59.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.393+0000 7f45a4473640 1 -- 172.21.15.78:0/2174481949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1006e0 msgr2=0x7f459c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:59.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.393+0000 7f45a4473640 1 --2- 172.21.15.78:0/2174481949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1006e0 0x7f459c100b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4594009ca0 tx=0x7f459402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- 172.21.15.78:0/2174481949 shutdown_connections 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 --2- 172.21.15.78:0/2174481949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f459c102290 0x7f459c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 --2- 172.21.15.78:0/2174481949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c1018d0 0x7f459c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 --2- 172.21.15.78:0/2174481949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1006e0 0x7f459c100b00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- 172.21.15.78:0/2174481949 >> 172.21.15.78:0/2174481949 conn(0x7f459c0fbc70 msgr2=0x7f459c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- 172.21.15.78:0/2174481949 shutdown_connections 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- 172.21.15.78:0/2174481949 wait complete. 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 Processor -- start 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- start start 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c1006e0 0x7f459c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f459c1018d0 0x7f459c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c102290 0x7f459c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f459c10c280 con 0x7f459c102290 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f459c10c100 con 0x7f459c1018d0 2024-03-16T11:03:59.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a4473640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f459c10c400 con 0x7f459c1006e0 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a2a10640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c102290 0x7f459c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a2a10640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c102290 0x7f459c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35926/0 (socket says 172.21.15.78:35926) 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a2a10640 1 -- 172.21.15.78:0/365540742 learned_addr learned my addr 172.21.15.78:0/365540742 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a220f640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c1006e0 0x7f459c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.394+0000 7f45a1a0e640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f459c1018d0 0x7f459c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a2a10640 1 -- 172.21.15.78:0/365540742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c1006e0 msgr2=0x7f459c0694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a2a10640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c1006e0 0x7f459c0694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a2a10640 1 -- 172.21.15.78:0/365540742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f459c1018d0 msgr2=0x7f459c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a2a10640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f459c1018d0 0x7f459c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a2a10640 1 -- 172.21.15.78:0/365540742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459c19f780 con 0x7f459c102290 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a2a10640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c102290 0x7f459c19ee70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4598009fd0 tx=0x7f459800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45937fe640 1 -- 172.21.15.78:0/365540742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598011830 con 0x7f459c102290 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459c19fa10 con 0x7f459c102290 2024-03-16T11:03:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45937fe640 1 -- 172.21.15.78:0/365540742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4598016070 con 0x7f459c102290 2024-03-16T11:03:59.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45937fe640 1 -- 172.21.15.78:0/365540742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598010de0 con 0x7f459c102290 2024-03-16T11:03:59.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.395+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f459c19ff90 con 0x7f459c102290 2024-03-16T11:03:59.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.396+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f459c107a70 con 0x7f459c102290 2024-03-16T11:03:59.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.397+0000 7f45937fe640 1 -- 172.21.15.78:0/365540742 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f45980119d0 con 0x7f459c102290 2024-03-16T11:03:59.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.397+0000 7f45937fe640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45880a2800 0x7f45880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:03:59.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.398+0000 7f45a220f640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45880a2800 0x7f45880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:03:59.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.398+0000 7f45937fe640 1 -- 172.21.15.78:0/365540742 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7879+0+0 (secure 0 0 0) 0x7f45980ce5c0 con 0x7f459c102290 2024-03-16T11:03:59.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.398+0000 7f45a220f640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45880a2800 0x7f45880a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4594032380 tx=0x7f4594000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:03:59.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.401+0000 7f45937fe640 1 -- 172.21.15.78:0/365540742 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f459809b810 con 0x7f459c102290 2024-03-16T11:03:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2974644507' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2974644507' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1889975738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:59.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:03:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2974644507' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2974644507' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1889975738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:59.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:59.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:03:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:03:59.615+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f459c062450 con 0x7f459c102290 2024-03-16T11:03:59.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2974644507' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2974644507' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1889975738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:03:59.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:03:59.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:03:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:04:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.278+0000 7f45937fe640 1 -- 172.21.15.78:0/365540742 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f459809b110 con 0x7f459c102290 2024-03-16T11:04:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T11:04:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.282+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45880a2800 msgr2=0x7f45880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.282+0000 7f45a4473640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45880a2800 0x7f45880a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4594032380 tx=0x7f4594000f40 comp rx=0 tx=0).stop 2024-03-16T11:04:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c102290 msgr2=0x7f459c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c102290 0x7f459c19ee70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4598009fd0 tx=0x7f459800b8e0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 shutdown_connections 2024-03-16T11:04:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c102290 0x7f459c19ee70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f459c1018d0 0x7f459c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f45880a2800 0x7f45880a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 --2- 172.21.15.78:0/365540742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c1006e0 0x7f459c0694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 >> 172.21.15.78:0/365540742 conn(0x7f459c0fbc70 msgr2=0x7f459c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 shutdown_connections 2024-03-16T11:04:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.283+0000 7f45a4473640 1 -- 172.21.15.78:0/365540742 wait complete. 2024-03-16T11:04:00.304 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.405+0000 7f83fc9a9640 1 -- 172.21.15.78:0/1452814593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f81022b0 msgr2=0x7f83f8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.405+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/1452814593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f81022b0 0x7f83f8108b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f83f0009ca0 tx=0x7f83f002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 -- 172.21.15.78:0/1452814593 shutdown_connections 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/1452814593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f81022b0 0x7f83f8108b40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/1452814593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f81018f0 0x7f83f8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/1452814593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f8100700 0x7f83f8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 -- 172.21.15.78:0/1452814593 >> 172.21.15.78:0/1452814593 conn(0x7f83f80fbc70 msgr2=0x7f83f80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 -- 172.21.15.78:0/1452814593 shutdown_connections 2024-03-16T11:04:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 -- 172.21.15.78:0/1452814593 wait complete. 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 Processor -- start 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 -- start start 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f8100700 0x7f83f81968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f81018f0 0x7f83f8196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f81022b0 0x7f83f819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f83f810c2a0 con 0x7f83f8100700 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.406+0000 7f83fc9a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83f810c120 con 0x7f83f81022b0 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83fc9a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83f810c420 con 0x7f83f81018f0 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f8100700 0x7f83f81968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f8100700 0x7f83f81968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35944/0 (socket says 172.21.15.78:35944) 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 -- 172.21.15.78:0/2361757715 learned_addr learned my addr 172.21.15.78:0/2361757715 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f6d9d640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f81022b0 0x7f83f819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f5d9b640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f81018f0 0x7f83f8196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 -- 172.21.15.78:0/2361757715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f81018f0 msgr2=0x7f83f8196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f81018f0 0x7f83f8196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 -- 172.21.15.78:0/2361757715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f81022b0 msgr2=0x7f83f819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f81022b0 0x7f83f819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 -- 172.21.15.78:0/2361757715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f819e500 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f5d9b640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f81018f0 0x7f83f8196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83f659c640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f8100700 0x7f83f81968e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00b450 tx=0x7f83ec00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83e77fe640 1 -- 172.21.15.78:0/2361757715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec015020 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83e77fe640 1 -- 172.21.15.78:0/2361757715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83ec011d40 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f819e7f0 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83e77fe640 1 -- 172.21.15.78:0/2361757715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec00f340 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.407+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f819ed70 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.408+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f81079c0 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.409+0000 7f83e77fe640 1 -- 172.21.15.78:0/2361757715 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f83ec018020 con 0x7f83f8100700 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.410+0000 7f83e77fe640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83e00a2800 0x7f83e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.410+0000 7f83f5d9b640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83e00a2800 0x7f83e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.410+0000 7f83e77fe640 1 -- 172.21.15.78:0/2361757715 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f83ec0ce400 con 0x7f83f8100700 2024-03-16T11:04:00.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.410+0000 7f83f5d9b640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83e00a2800 0x7f83e00a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f83f8197df0 tx=0x7f83e800a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:00.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.414+0000 7f83e77fe640 1 -- 172.21.15.78:0/2361757715 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f83ec01f020 con 0x7f83f8100700 2024-03-16T11:04:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:00 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1889975738' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T11:04:00.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:00 smithi153 ceph-mon[35521]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T11:04:00.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:00 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/365540742' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T11:04:00.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:00 smithi153 ceph-mon[35521]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T11:04:00.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:00 smithi153 ceph-mon[35521]: pgmap v285: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:04:00.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:00 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1889975738' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T11:04:00.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:00 smithi078 ceph-mon[32235]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T11:04:00.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:00 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/365540742' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T11:04:00.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:00 smithi078 ceph-mon[32235]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T11:04:00.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:00 smithi078 ceph-mon[32235]: pgmap v285: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:04:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.592+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83f80623e0 con 0x7f83f8100700 2024-03-16T11:04:00.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.594+0000 7f83e77fe640 1 -- 172.21.15.78:0/2361757715 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22149 (secure 0 0 0) 0x7f83ec00d040 con 0x7f83f8100700 2024-03-16T11:04:00.594 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:00.595 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:03:59.616869+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:04:00.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83e00a2800 msgr2=0x7f83e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83e00a2800 0x7f83e00a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f83f8197df0 tx=0x7f83e800a5c0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f8100700 msgr2=0x7f83f81968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f8100700 0x7f83f81968e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00b450 tx=0x7f83ec00b910 comp rx=0 tx=0).stop 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 shutdown_connections 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83f81022b0 0x7f83f819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83e00a2800 0x7f83e00a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f81018f0 0x7f83f8196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 --2- 172.21.15.78:0/2361757715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83f8100700 0x7f83f81968e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 >> 172.21.15.78:0/2361757715 conn(0x7f83f80fbc70 msgr2=0x7f83f8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 shutdown_connections 2024-03-16T11:04:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.596+0000 7f83fc9a9640 1 -- 172.21.15.78:0/2361757715 wait complete. 2024-03-16T11:04:00.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:04:00.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.708+0000 7f671f6c6640 1 -- 172.21.15.78:0/3556415170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 msgr2=0x7f6718108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.708+0000 7f671f6c6640 1 --2- 172.21.15.78:0/3556415170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718108b40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6710009ca0 tx=0x7f671002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- 172.21.15.78:0/3556415170 shutdown_connections 2024-03-16T11:04:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 --2- 172.21.15.78:0/3556415170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718108b40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 --2- 172.21.15.78:0/3556415170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67181018f0 0x7f6718101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 --2- 172.21.15.78:0/3556415170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6718100700 0x7f6718100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- 172.21.15.78:0/3556415170 >> 172.21.15.78:0/3556415170 conn(0x7f67180fbc70 msgr2=0x7f67180fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- 172.21.15.78:0/3556415170 shutdown_connections 2024-03-16T11:04:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- 172.21.15.78:0/3556415170 wait complete. 2024-03-16T11:04:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 Processor -- start 2024-03-16T11:04:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- start start 2024-03-16T11:04:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6718100700 0x7f6718104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67181018f0 0x7f6718102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f671810c320 con 0x7f67181022b0 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f671810c1a0 con 0x7f67181018f0 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671f6c6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f671810c4a0 con 0x7f6718100700 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671dc63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671dc63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35962/0 (socket says 172.21.15.78:35962) 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.709+0000 7f671dc63640 1 -- 172.21.15.78:0/2357757974 learned_addr learned my addr 172.21.15.78:0/2357757974 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671cc61640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67181018f0 0x7f6718102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671d462640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6718100700 0x7f6718104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671dc63640 1 -- 172.21.15.78:0/2357757974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6718100700 msgr2=0x7f6718104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671dc63640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6718100700 0x7f6718104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671dc63640 1 -- 172.21.15.78:0/2357757974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67181018f0 msgr2=0x7f6718102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671dc63640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67181018f0 0x7f6718102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671dc63640 1 -- 172.21.15.78:0/2357757974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6718103aa0 con 0x7f67181022b0 2024-03-16T11:04:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671d462640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6718100700 0x7f6718104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:04:00.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671dc63640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718103210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6710037040 tx=0x7f67100328f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:00.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f670e7fc640 1 -- 172.21.15.78:0/2357757974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710041020 con 0x7f67181022b0 2024-03-16T11:04:00.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67181a1c70 con 0x7f67181022b0 2024-03-16T11:04:00.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f670e7fc640 1 -- 172.21.15.78:0/2357757974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6710003e20 con 0x7f67181022b0 2024-03-16T11:04:00.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f670e7fc640 1 -- 172.21.15.78:0/2357757974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710039040 con 0x7f67181022b0 2024-03-16T11:04:00.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.710+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67181a21c0 con 0x7f67181022b0 2024-03-16T11:04:00.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.711+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67181a23f0 con 0x7f67181022b0 2024-03-16T11:04:00.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:00 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1889975738' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T11:04:00.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:00 smithi098 ceph-mon[35848]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T11:04:00.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:00 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/365540742' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T11:04:00.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:00 smithi098 ceph-mon[35848]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T11:04:00.716 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:00 smithi098 ceph-mon[35848]: pgmap v285: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:04:00.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.712+0000 7f670e7fc640 1 -- 172.21.15.78:0/2357757974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6710044020 con 0x7f67181022b0 2024-03-16T11:04:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.712+0000 7f670e7fc640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f67040a2880 0x7f67040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.713+0000 7f671d462640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f67040a2880 0x7f67040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.713+0000 7f670e7fc640 1 -- 172.21.15.78:0/2357757974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f67100f0f80 con 0x7f67181022b0 2024-03-16T11:04:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.713+0000 7f671d462640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f67040a2880 0x7f67040a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6708006fd0 tx=0x7f6708008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.717+0000 7f670e7fc640 1 -- 172.21.15.78:0/2357757974 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f671004b020 con 0x7f67181022b0 2024-03-16T11:04:00.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.892+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67181a2780 con 0x7f67181022b0 2024-03-16T11:04:00.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.894+0000 7f670e7fc640 1 -- 172.21.15.78:0/2357757974 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22149 (secure 0 0 0) 0x7f67181a2780 con 0x7f67181022b0 2024-03-16T11:04:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:00.895 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:03:59.616869+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f67040a2880 msgr2=0x7f67040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f67040a2880 0x7f67040a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6708006fd0 tx=0x7f6708008040 comp rx=0 tx=0).stop 2024-03-16T11:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 msgr2=0x7f6718103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718103210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6710037040 tx=0x7f67100328f0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 shutdown_connections 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67181022b0 0x7f6718103210 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67181018f0 0x7f6718102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f67040a2880 0x7f67040a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 --2- 172.21.15.78:0/2357757974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6718100700 0x7f6718104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 >> 172.21.15.78:0/2357757974 conn(0x7f67180fbc70 msgr2=0x7f6718068990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 shutdown_connections 2024-03-16T11:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:00.896+0000 7f671f6c6640 1 -- 172.21.15.78:0/2357757974 wait complete. 2024-03-16T11:04:00.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 -- 172.21.15.78:0/4091892332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41022a0 msgr2=0x7f29e4108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/4091892332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41022a0 0x7f29e4108b30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f29dc009ca0 tx=0x7f29dc02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 -- 172.21.15.78:0/4091892332 shutdown_connections 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/4091892332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41022a0 0x7f29e4108b30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/4091892332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e41018e0 0x7f29e4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/4091892332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29e41006f0 0x7f29e4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 -- 172.21.15.78:0/4091892332 >> 172.21.15.78:0/4091892332 conn(0x7f29e40fbc80 msgr2=0x7f29e40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 -- 172.21.15.78:0/4091892332 shutdown_connections 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 -- 172.21.15.78:0/4091892332 wait complete. 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 Processor -- start 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.014+0000 7f29e9ddf640 1 -- start start 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29e41006f0 0x7f29e4196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41018e0 0x7f29e4196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e41022a0 0x7f29e419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f29e410c290 con 0x7f29e41018e0 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29e410c110 con 0x7f29e41022a0 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29e410c410 con 0x7f29e41006f0 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41018e0 0x7f29e4196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41018e0 0x7f29e4196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35984/0 (socket says 172.21.15.78:35984) 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 -- 172.21.15.78:0/3537419135 learned_addr learned my addr 172.21.15.78:0/3537419135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e3fff640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e41022a0 0x7f29e419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 -- 172.21.15.78:0/3537419135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29e41006f0 msgr2=0x7f29e4196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29e41006f0 0x7f29e4196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 -- 172.21.15.78:0/3537419135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e41022a0 msgr2=0x7f29e419def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e41022a0 0x7f29e419def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 -- 172.21.15.78:0/3537419135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e419e560 con 0x7f29e41018e0 2024-03-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e2ffd640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41018e0 0x7f29e4196e80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c890 tx=0x7f29d800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e0ff9640 1 -- 172.21.15.78:0/3537419135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8004220 con 0x7f29e41018e0 2024-03-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e0ff9640 1 -- 172.21.15.78:0/3537419135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29d80043c0 con 0x7f29e41018e0 2024-03-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e419e850 con 0x7f29e41018e0 2024-03-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e0ff9640 1 -- 172.21.15.78:0/3537419135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8004a00 con 0x7f29e41018e0 2024-03-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.015+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e419ee20 con 0x7f29e41018e0 2024-03-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.016+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29e4107a80 con 0x7f29e41018e0 2024-03-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.017+0000 7f29e0ff9640 1 -- 172.21.15.78:0/3537419135 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f29d8004ba0 con 0x7f29e41018e0 2024-03-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.018+0000 7f29e0ff9640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f29cc0a2800 0x7f29cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.018+0000 7f29e37fe640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f29cc0a2800 0x7f29cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.018+0000 7f29e0ff9640 1 -- 172.21.15.78:0/3537419135 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f29d80ceab0 con 0x7f29e41018e0 2024-03-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.019+0000 7f29e37fe640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f29cc0a2800 0x7f29cc0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f29e4101740 tx=0x7f29d4006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.022+0000 7f29e0ff9640 1 -- 172.21.15.78:0/3537419135 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29d8014030 con 0x7f29e41018e0 2024-03-16T11:04:01.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.234+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f29e402d1b0 con 0x7f29e41018e0 2024-03-16T11:04:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.279+0000 7f29e0ff9640 1 -- 172.21.15.78:0/3537419135 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f29d809bca0 con 0x7f29e41018e0 2024-03-16T11:04:01.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f29cc0a2800 msgr2=0x7f29cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f29cc0a2800 0x7f29cc0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f29e4101740 tx=0x7f29d4006c90 comp rx=0 tx=0).stop 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41018e0 msgr2=0x7f29e4196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41018e0 0x7f29e4196e80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c890 tx=0x7f29d800cd50 comp rx=0 tx=0).stop 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 shutdown_connections 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e41022a0 0x7f29e419def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29e41018e0 0x7f29e4196e80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f29cc0a2800 0x7f29cc0a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 --2- 172.21.15.78:0/3537419135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29e41006f0 0x7f29e4196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 >> 172.21.15.78:0/3537419135 conn(0x7f29e40fbc80 msgr2=0x7f29e4104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 shutdown_connections 2024-03-16T11:04:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.282+0000 7f29e9ddf640 1 -- 172.21.15.78:0/3537419135 wait complete. 2024-03-16T11:04:01.296 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-16T11:04:01.296 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 -- 172.21.15.78:0/698969293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 msgr2=0x7fb35c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 --2- 172.21.15.78:0/698969293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c102e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb344009900 tx=0x7fb34402bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 -- 172.21.15.78:0/698969293 shutdown_connections 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 --2- 172.21.15.78:0/698969293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb35c1033a0 0x7fb35c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 --2- 172.21.15.78:0/698969293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c102e60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 --2- 172.21.15.78:0/698969293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35c1017f0 0x7fb35c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 -- 172.21.15.78:0/698969293 >> 172.21.15.78:0/698969293 conn(0x7fb35c0fcd80 msgr2=0x7fb35c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 -- 172.21.15.78:0/698969293 shutdown_connections 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 -- 172.21.15.78:0/698969293 wait complete. 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.399+0000 7fb360f59640 1 Processor -- start 2024-03-16T11:04:01.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 -- start start 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb35c1017f0 0x7fb35c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35c1033a0 0x7fb35c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c10d390 con 0x7fb35c1029e0 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c10d210 con 0x7fb35c1017f0 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c10d510 con 0x7fb35c1033a0 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35998/0 (socket says 172.21.15.78:35998) 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 -- 172.21.15.78:0/2857203241 learned_addr learned my addr 172.21.15.78:0/2857203241 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb35a59c640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb35c1017f0 0x7fb35c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 -- 172.21.15.78:0/2857203241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35c1033a0 msgr2=0x7fb35c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35c1033a0 0x7fb35c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 -- 172.21.15.78:0/2857203241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb35c1017f0 msgr2=0x7fb35c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb35c1017f0 0x7fb35c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 -- 172.21.15.78:0/2857203241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c19f700 con 0x7fb35c1029e0 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb353fff640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c198020 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb344006010 tx=0x7fb344004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb3537fe640 1 -- 172.21.15.78:0/2857203241 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344043020 con 0x7fb35c1029e0 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb3537fe640 1 -- 172.21.15.78:0/2857203241 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb344004440 con 0x7fb35c1029e0 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35c19f990 con 0x7fb35c1029e0 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb3537fe640 1 -- 172.21.15.78:0/2857203241 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344003a80 con 0x7fb35c1029e0 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.400+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35c19ff60 con 0x7fb35c1029e0 2024-03-16T11:04:01.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.401+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35c108bd0 con 0x7fb35c1029e0 2024-03-16T11:04:01.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.402+0000 7fb3537fe640 1 -- 172.21.15.78:0/2857203241 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb344046020 con 0x7fb35c1029e0 2024-03-16T11:04:01.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.403+0000 7fb3537fe640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3480a2800 0x7fb3480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:01.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.403+0000 7fb35a59c640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3480a2800 0x7fb3480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:01.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.403+0000 7fb3537fe640 1 -- 172.21.15.78:0/2857203241 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fb3440f0c50 con 0x7fb35c1029e0 2024-03-16T11:04:01.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.403+0000 7fb35a59c640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3480a2800 0x7fb3480a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb35c102840 tx=0x7fb34c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:01.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.407+0000 7fb3537fe640 1 -- 172.21.15.78:0/2857203241 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb3440f6020 con 0x7fb35c1029e0 2024-03-16T11:04:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T11:04:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T11:04:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/365540742' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T11:04:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T11:04:01.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T11:04:01.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2361757715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:01.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2357757974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:01.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:01 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3537419135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T11:04:01.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T11:04:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T11:04:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/365540742' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T11:04:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T11:04:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T11:04:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2361757715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2357757974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:01 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3537419135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T11:04:01.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:01.651+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fb35c02d400 con 0x7fb35c1029e0 2024-03-16T11:04:01.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T11:04:01.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T11:04:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/365540742' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T11:04:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T11:04:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T11:04:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2361757715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2357757974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:01.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:01 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3537419135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T11:04:02.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.290+0000 7fb3537fe640 1 -- 172.21.15.78:0/2857203241 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7fb34404d020 con 0x7fb35c1029e0 2024-03-16T11:04:02.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3480a2800 msgr2=0x7fb3480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3480a2800 0x7fb3480a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb35c102840 tx=0x7fb34c009110 comp rx=0 tx=0).stop 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 msgr2=0x7fb35c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c198020 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb344006010 tx=0x7fb344004040 comp rx=0 tx=0).stop 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 shutdown_connections 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3480a2800 0x7fb3480a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35c1033a0 0x7fb35c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb35c1029e0 0x7fb35c198020 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 --2- 172.21.15.78:0/2857203241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb35c1017f0 0x7fb35c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 >> 172.21.15.78:0/2857203241 conn(0x7fb35c0fcd80 msgr2=0x7fb35c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 shutdown_connections 2024-03-16T11:04:02.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.293+0000 7fb360f59640 1 -- 172.21.15.78:0/2857203241 wait complete. 2024-03-16T11:04:02.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-16T11:04:02.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 -- 172.21.15.78:0/1033971259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 msgr2=0x7fd18c1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:02.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 --2- 172.21.15.78:0/1033971259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c1097f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd184009ca0 tx=0x7fd18402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 -- 172.21.15.78:0/1033971259 shutdown_connections 2024-03-16T11:04:02.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 --2- 172.21.15.78:0/1033971259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c1097f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 --2- 172.21.15.78:0/1033971259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c104a70 0x7fd18c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 --2- 172.21.15.78:0/1033971259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd18c102140 0x7fd18c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 -- 172.21.15.78:0/1033971259 >> 172.21.15.78:0/1033971259 conn(0x7fd18c0fbb50 msgr2=0x7fd18c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 -- 172.21.15.78:0/1033971259 shutdown_connections 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.416+0000 7fd18bfff640 1 -- 172.21.15.78:0/1033971259 wait complete. 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 Processor -- start 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 -- start start 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102140 0x7fd18c1978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd18c104a70 0x7fd18c197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c10d260 con 0x7fd18c1073a0 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c10d0e0 con 0x7fd18c104a70 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18bfff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c10d3e0 con 0x7fd18c102140 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c19ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36020/0 (socket says 172.21.15.78:36020) 2024-03-16T11:04:02.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 -- 172.21.15.78:0/3214034094 learned_addr learned my addr 172.21.15.78:0/3214034094 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd183fff640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd18c104a70 0x7fd18c197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18affd640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102140 0x7fd18c1978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 -- 172.21.15.78:0/3214034094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102140 msgr2=0x7fd18c1978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102140 0x7fd18c1978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 -- 172.21.15.78:0/3214034094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd18c104a70 msgr2=0x7fd18c197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd18c104a70 0x7fd18c197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.417+0000 7fd18b7fe640 1 -- 172.21.15.78:0/3214034094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18c19f500 con 0x7fd18c1073a0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.418+0000 7fd18b7fe640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c19ee90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd18402c700 tx=0x7fd1840328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.418+0000 7fd1909fd640 1 -- 172.21.15.78:0/3214034094 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd184041020 con 0x7fd18c1073a0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.418+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18c19f790 con 0x7fd18c1073a0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.418+0000 7fd1909fd640 1 -- 172.21.15.78:0/3214034094 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd184003e20 con 0x7fd18c1073a0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.418+0000 7fd1909fd640 1 -- 172.21.15.78:0/3214034094 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd184039040 con 0x7fd18c1073a0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.418+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd18c19fd00 con 0x7fd18c1073a0 2024-03-16T11:04:02.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.419+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd18c0fd640 con 0x7fd18c1073a0 2024-03-16T11:04:02.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.420+0000 7fd1909fd640 1 -- 172.21.15.78:0/3214034094 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd184044020 con 0x7fd18c1073a0 2024-03-16T11:04:02.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.420+0000 7fd1909fd640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd1780a2800 0x7fd1780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:02.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.420+0000 7fd18affd640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd1780a2800 0x7fd1780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:02.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.420+0000 7fd1909fd640 1 -- 172.21.15.78:0/3214034094 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fd1840f0fe0 con 0x7fd18c1073a0 2024-03-16T11:04:02.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.421+0000 7fd18affd640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd1780a2800 0x7fd1780a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009b60 tx=0x7fd17c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:02.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.424+0000 7fd1909fd640 1 -- 172.21.15.78:0/3214034094 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd18404b020 con 0x7fd18c1073a0 2024-03-16T11:04:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:02 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3537419135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T11:04:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:02 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] up:active 2024-03-16T11:04:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:02 smithi153 ceph-mon[35521]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:04:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:02 smithi153 ceph-mon[35521]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T11:04:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:02 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2857203241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T11:04:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:02 smithi153 ceph-mon[35521]: pgmap v288: 97 pgs: 23 creating+peering, 21 active+clean, 53 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 1 op/s 2024-03-16T11:04:02.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:02 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3537419135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T11:04:02.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:02 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] up:active 2024-03-16T11:04:02.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:02 smithi078 ceph-mon[32235]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:04:02.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:02 smithi078 ceph-mon[32235]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T11:04:02.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:02 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2857203241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T11:04:02.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:02 smithi078 ceph-mon[32235]: pgmap v288: 97 pgs: 23 creating+peering, 21 active+clean, 53 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 1 op/s 2024-03-16T11:04:02.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:02.638+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fd18c02d1b0 con 0x7fd18c1073a0 2024-03-16T11:04:02.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:02 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3537419135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T11:04:02.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:02 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] up:active 2024-03-16T11:04:02.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:02 smithi098 ceph-mon[35848]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:04:02.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:02 smithi098 ceph-mon[35848]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T11:04:02.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:02 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2857203241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T11:04:02.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:02 smithi098 ceph-mon[35848]: pgmap v288: 97 pgs: 23 creating+peering, 21 active+clean, 53 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 1 op/s 2024-03-16T11:04:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.298+0000 7fd1909fd640 1 -- 172.21.15.78:0/3214034094 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7fd1840333a0 con 0x7fd18c1073a0 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd1780a2800 msgr2=0x7fd1780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd1780a2800 0x7fd1780a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009b60 tx=0x7fd17c005f90 comp rx=0 tx=0).stop 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 msgr2=0x7fd18c19ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c19ee90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd18402c700 tx=0x7fd1840328a0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 shutdown_connections 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd18c1073a0 0x7fd18c19ee90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd18c104a70 0x7fd18c197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd1780a2800 0x7fd1780a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 --2- 172.21.15.78:0/3214034094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102140 0x7fd18c1978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 >> 172.21.15.78:0/3214034094 conn(0x7fd18c0fbb50 msgr2=0x7fd18c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:03.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 shutdown_connections 2024-03-16T11:04:03.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.301+0000 7fd18bfff640 1 -- 172.21.15.78:0/3214034094 wait complete. 2024-03-16T11:04:03.317 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T11:04:03.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 -- 172.21.15.78:0/1890561249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3081018d0 msgr2=0x7fb308101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:03.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 --2- 172.21.15.78:0/1890561249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3081018d0 0x7fb308101d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009ca0 tx=0x7fb2f802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 -- 172.21.15.78:0/1890561249 shutdown_connections 2024-03-16T11:04:03.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 --2- 172.21.15.78:0/1890561249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb308102290 0x7fb308108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 --2- 172.21.15.78:0/1890561249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3081018d0 0x7fb308101d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 --2- 172.21.15.78:0/1890561249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3081006e0 0x7fb308100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 -- 172.21.15.78:0/1890561249 >> 172.21.15.78:0/1890561249 conn(0x7fb3080fbc70 msgr2=0x7fb3080fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 -- 172.21.15.78:0/1890561249 shutdown_connections 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.423+0000 7fb30eb77640 1 -- 172.21.15.78:0/1890561249 wait complete. 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 Processor -- start 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 -- start start 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3081006e0 0x7fb308197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3081018d0 0x7fb308197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb308102290 0x7fb30819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb30810d330 con 0x7fb308102290 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb30810d1b0 con 0x7fb3081018d0 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30eb77640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb30810d4b0 con 0x7fb3081006e0 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb308102290 0x7fb30819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb308102290 0x7fb30819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36046/0 (socket says 172.21.15.78:36046) 2024-03-16T11:04:03.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 -- 172.21.15.78:0/218216407 learned_addr learned my addr 172.21.15.78:0/218216407 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30c913640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3081006e0 0x7fb308197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb307fff640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3081018d0 0x7fb308197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 -- 172.21.15.78:0/218216407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3081006e0 msgr2=0x7fb308197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3081006e0 0x7fb308197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 -- 172.21.15.78:0/218216407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3081018d0 msgr2=0x7fb308197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3081018d0 0x7fb308197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.424+0000 7fb30d114640 1 -- 172.21.15.78:0/218216407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30819f660 con 0x7fb308102290 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.425+0000 7fb30d114640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb308102290 0x7fb30819eff0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e970 tx=0x7fb30000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.425+0000 7fb305ffb640 1 -- 172.21.15.78:0/218216407 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30000ccc0 con 0x7fb308102290 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.425+0000 7fb305ffb640 1 -- 172.21.15.78:0/218216407 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb30000ce60 con 0x7fb308102290 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.425+0000 7fb305ffb640 1 -- 172.21.15.78:0/218216407 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300010430 con 0x7fb308102290 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.425+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30819f950 con 0x7fb308102290 2024-03-16T11:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.425+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30819fe70 con 0x7fb308102290 2024-03-16T11:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.426+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb308107b20 con 0x7fb308102290 2024-03-16T11:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.427+0000 7fb305ffb640 1 -- 172.21.15.78:0/218216407 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb300004ff0 con 0x7fb308102290 2024-03-16T11:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.427+0000 7fb305ffb640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb2f40a2880 0x7fb2f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.428+0000 7fb30c913640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb2f40a2880 0x7fb2f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.428+0000 7fb305ffb640 1 -- 172.21.15.78:0/218216407 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fb3000ceca0 con 0x7fb308102290 2024-03-16T11:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.428+0000 7fb30c913640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb2f40a2880 0x7fb2f40a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb308101730 tx=0x7fb2f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.432+0000 7fb305ffb640 1 -- 172.21.15.78:0/218216407 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb30009be90 con 0x7fb308102290 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: Cluster is now healthy 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2857203241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T11:04:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T11:04:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:03 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3214034094' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: Cluster is now healthy 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2857203241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T11:04:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:03 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3214034094' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T11:04:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:03.656+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb3080623e0 con 0x7fb308102290 2024-03-16T11:04:03.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T11:04:03.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: Cluster is now healthy 2024-03-16T11:04:03.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T11:04:03.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2857203241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T11:04:03.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T11:04:03.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:04:03.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T11:04:03.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T11:04:03.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:03 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3214034094' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T11:04:04.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.476+0000 7fb305ffb640 1 -- 172.21.15.78:0/218216407 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7fb30009b790 con 0x7fb308102290 2024-03-16T11:04:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb2f40a2880 msgr2=0x7fb2f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb2f40a2880 0x7fb2f40a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb308101730 tx=0x7fb2f0005f90 comp rx=0 tx=0).stop 2024-03-16T11:04:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb308102290 msgr2=0x7fb30819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb308102290 0x7fb30819eff0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e970 tx=0x7fb30000ed70 comp rx=0 tx=0).stop 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 shutdown_connections 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb308102290 0x7fb30819eff0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3081018d0 0x7fb308197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb2f40a2880 0x7fb2f40a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 --2- 172.21.15.78:0/218216407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3081006e0 0x7fb308197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 >> 172.21.15.78:0/218216407 conn(0x7fb3080fbc70 msgr2=0x7fb308104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 shutdown_connections 2024-03-16T11:04:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.479+0000 7fb30eb77640 1 -- 172.21.15.78:0/218216407 wait complete. 2024-03-16T11:04:04.494 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-16T11:04:04.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-16T11:04:04.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3214034094' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T11:04:04.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] up:active 2024-03-16T11:04:04.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T11:04:04.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T11:04:04.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: Cluster is now healthy 2024-03-16T11:04:04.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T11:04:04.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T11:04:04.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T11:04:04.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] up:active 2024-03-16T11:04:04.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:04:04.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/218216407' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T11:04:04.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-mon[32235]: pgmap v289: 97 pgs: 20 creating+peering, 68 active+clean, 9 unknown; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 179 B/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:04:04.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- 172.21.15.78:0/3835856871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 msgr2=0x7f5dc8101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 --2- 172.21.15.78:0/3835856871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8101d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009900 tx=0x7f5db802bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- 172.21.15.78:0/3835856871 shutdown_connections 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 --2- 172.21.15.78:0/3835856871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc8102290 0x7f5dc8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 --2- 172.21.15.78:0/3835856871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8101d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 --2- 172.21.15.78:0/3835856871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc81006e0 0x7f5dc8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- 172.21.15.78:0/3835856871 >> 172.21.15.78:0/3835856871 conn(0x7f5dc80fbc70 msgr2=0x7f5dc80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- 172.21.15.78:0/3835856871 shutdown_connections 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- 172.21.15.78:0/3835856871 wait complete. 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 Processor -- start 2024-03-16T11:04:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- start start 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc81006e0 0x7f5dc8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8102290 0x7f5dc819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc810c210 con 0x7f5dc81018d0 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc810c090 con 0x7f5dc8102290 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.604+0000 7f5dcf562640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc810c390 con 0x7f5dc81006e0 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36068/0 (socket says 172.21.15.78:36068) 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 -- 172.21.15.78:0/4044926584 learned_addr learned my addr 172.21.15.78:0/4044926584 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dcd2fe640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc81006e0 0x7f5dc8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dcdaff640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8102290 0x7f5dc819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 -- 172.21.15.78:0/4044926584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc81006e0 msgr2=0x7f5dc8197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc81006e0 0x7f5dc8197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 -- 172.21.15.78:0/4044926584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8102290 msgr2=0x7f5dc819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8102290 0x7f5dc819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 -- 172.21.15.78:0/4044926584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc819f680 con 0x7f5dc81018d0 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dccafd640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8197fa0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5db8006010 tx=0x7f5db8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dbe7fc640 1 -- 172.21.15.78:0/4044926584 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8043020 con 0x7f5dc81018d0 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc819f910 con 0x7f5dc81018d0 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dbe7fc640 1 -- 172.21.15.78:0/4044926584 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5db80043d0 con 0x7f5dc81018d0 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dbe7fc640 1 -- 172.21.15.78:0/4044926584 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8003b80 con 0x7f5dc81018d0 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.605+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc819feb0 con 0x7f5dc81018d0 2024-03-16T11:04:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.606+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc81a01a0 con 0x7f5dc81018d0 2024-03-16T11:04:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.607+0000 7f5dbe7fc640 1 -- 172.21.15.78:0/4044926584 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5db8046020 con 0x7f5dc81018d0 2024-03-16T11:04:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.608+0000 7f5dbe7fc640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5db40a2800 0x7f5db40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.608+0000 7f5dcd2fe640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5db40a2800 0x7f5db40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.608+0000 7f5dbe7fc640 1 -- 172.21.15.78:0/4044926584 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f5db80f1130 con 0x7f5dc81018d0 2024-03-16T11:04:04.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.609+0000 7f5dcd2fe640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5db40a2800 0x7f5db40a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8101730 tx=0x7f5dc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:04.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.612+0000 7f5dbe7fc640 1 -- 172.21.15.78:0/4044926584 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5db804d020 con 0x7f5dc81018d0 2024-03-16T11:04:04.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3214034094' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T11:04:04.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] up:active 2024-03-16T11:04:04.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: Cluster is now healthy 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] up:active 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/218216407' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T11:04:04.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:04 smithi098 ceph-mon[35848]: pgmap v289: 97 pgs: 20 creating+peering, 68 active+clean, 9 unknown; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 179 B/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:04:04.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3214034094' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T11:04:04.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] up:active 2024-03-16T11:04:04.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T11:04:04.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: Cluster is now healthy 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] up:active 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/218216407' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T11:04:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:04 smithi153 ceph-mon[35521]: pgmap v289: 97 pgs: 20 creating+peering, 68 active+clean, 9 unknown; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 179 B/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:04:04.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.814+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5dc8069050 con 0x7f5db40a2800 2024-03-16T11:04:04.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.836+0000 7f5dbe7fc640 1 -- 172.21.15.78:0/4044926584 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5dc8069050 con 0x7f5db40a2800 2024-03-16T11:04:04.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5db40a2800 msgr2=0x7f5db40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5db40a2800 0x7f5db40a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8101730 tx=0x7f5dc4005f90 comp rx=0 tx=0).stop 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 msgr2=0x7f5dc8197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8197fa0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5db8006010 tx=0x7f5db8005e40 comp rx=0 tx=0).stop 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 shutdown_connections 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dc8102290 0x7f5dc819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc81018d0 0x7f5dc8197fa0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5db40a2800 0x7f5db40a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 --2- 172.21.15.78:0/4044926584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc81006e0 0x7f5dc8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 >> 172.21.15.78:0/4044926584 conn(0x7f5dc80fbc70 msgr2=0x7f5dc8068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 shutdown_connections 2024-03-16T11:04:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.838+0000 7f5dcf562640 1 -- 172.21.15.78:0/4044926584 wait complete. 2024-03-16T11:04:04.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-16T11:04:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 -- 172.21.15.78:0/3327560370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e341009c0 msgr2=0x7f0e34100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 --2- 172.21.15.78:0/3327560370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e341009c0 0x7f0e34100e40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009900 tx=0x7f0e2402bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 -- 172.21.15.78:0/3327560370 shutdown_connections 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 --2- 172.21.15.78:0/3327560370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e34101380 0x7f0e34108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 --2- 172.21.15.78:0/3327560370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e341009c0 0x7f0e34100e40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 --2- 172.21.15.78:0/3327560370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e34101c90 0x7f0e341020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 -- 172.21.15.78:0/3327560370 >> 172.21.15.78:0/3327560370 conn(0x7f0e340fb410 msgr2=0x7f0e340fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 -- 172.21.15.78:0/3327560370 shutdown_connections 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 -- 172.21.15.78:0/3327560370 wait complete. 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 Processor -- start 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.956+0000 7f0e38f37640 1 -- start start 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e341009c0 0x7f0e34197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e34101380 0x7f0e34197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34101c90 0x7f0e3419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3410d360 con 0x7f0e34101c90 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3410d1e0 con 0x7f0e341009c0 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3410d4e0 con 0x7f0e34101380 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34101c90 0x7f0e3419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34101c90 0x7f0e3419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36086/0 (socket says 172.21.15.78:36086) 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 -- 172.21.15.78:0/4118326026 learned_addr learned my addr 172.21.15.78:0/4118326026 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e31d9b640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e34101380 0x7f0e34197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e3259c640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e341009c0 0x7f0e34197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 -- 172.21.15.78:0/4118326026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e34101380 msgr2=0x7f0e34197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e34101380 0x7f0e34197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 -- 172.21.15.78:0/4118326026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e341009c0 msgr2=0x7f0e34197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e341009c0 0x7f0e34197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 -- 172.21.15.78:0/4118326026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3419f660 con 0x7f0e34101c90 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e32d9d640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34101c90 0x7f0e3419eff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00eb10 tx=0x7f0e2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e237fe640 1 -- 172.21.15.78:0/4118326026 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c004b20 con 0x7f0e34101c90 2024-03-16T11:04:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3419f950 con 0x7f0e34101c90 2024-03-16T11:04:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e237fe640 1 -- 172.21.15.78:0/4118326026 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e2c004cc0 con 0x7f0e34101c90 2024-03-16T11:04:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e237fe640 1 -- 172.21.15.78:0/4118326026 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c010690 con 0x7f0e34101c90 2024-03-16T11:04:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.957+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3419fe70 con 0x7f0e34101c90 2024-03-16T11:04:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.958+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e34107bd0 con 0x7f0e34101c90 2024-03-16T11:04:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.959+0000 7f0e237fe640 1 -- 172.21.15.78:0/4118326026 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0e2c005000 con 0x7f0e34101c90 2024-03-16T11:04:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.960+0000 7f0e237fe640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e1c0a2800 0x7f0e1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.960+0000 7f0e3259c640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e1c0a2800 0x7f0e1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.960+0000 7f0e237fe640 1 -- 172.21.15.78:0/4118326026 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f0e2c0cf0a0 con 0x7f0e34101c90 2024-03-16T11:04:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.960+0000 7f0e3259c640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e1c0a2800 0x7f0e1c0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0e34069bb0 tx=0x7f0e28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:04.963+0000 7f0e237fe640 1 -- 172.21.15.78:0/4118326026 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e2c053740 con 0x7f0e34101c90 2024-03-16T11:04:05.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.168+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0e34061c20 con 0x7f0e1c0a2800 2024-03-16T11:04:05.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.191+0000 7f0e237fe640 1 -- 172.21.15.78:0/4118326026 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0e34061c20 con 0x7f0e1c0a2800 2024-03-16T11:04:05.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e1c0a2800 msgr2=0x7f0e1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e1c0a2800 0x7f0e1c0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0e34069bb0 tx=0x7f0e28005f90 comp rx=0 tx=0).stop 2024-03-16T11:04:05.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34101c90 msgr2=0x7f0e3419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34101c90 0x7f0e3419eff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00eb10 tx=0x7f0e2c00c2d0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 shutdown_connections 2024-03-16T11:04:05.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34101c90 0x7f0e3419eff0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e1c0a2800 0x7f0e1c0a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e34101380 0x7f0e34197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 --2- 172.21.15.78:0/4118326026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e341009c0 0x7f0e34197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 >> 172.21.15.78:0/4118326026 conn(0x7f0e340fb410 msgr2=0x7f0e34104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:05.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 shutdown_connections 2024-03-16T11:04:05.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.193+0000 7f0e38f37640 1 -- 172.21.15.78:0/4118326026 wait complete. 2024-03-16T11:04:05.204 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:04:04.818+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:04:05.204 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:04:04.818+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:04:05.204 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:04:04.818+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:04:05.204 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:04:04.818+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:04:05.204 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:04:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:04:04.818+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:04:05.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-16T11:04:05.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 -- 172.21.15.78:0/1203812629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc102290 msgr2=0x7feabc108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 --2- 172.21.15.78:0/1203812629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc102290 0x7feabc108b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feab4009900 tx=0x7feab402bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 -- 172.21.15.78:0/1203812629 shutdown_connections 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 --2- 172.21.15.78:0/1203812629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc102290 0x7feabc108b20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 --2- 172.21.15.78:0/1203812629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feabc1018d0 0x7feabc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 --2- 172.21.15.78:0/1203812629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feabc1006e0 0x7feabc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 -- 172.21.15.78:0/1203812629 >> 172.21.15.78:0/1203812629 conn(0x7feabc0fbc70 msgr2=0x7feabc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 -- 172.21.15.78:0/1203812629 shutdown_connections 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.311+0000 7feac1631640 1 -- 172.21.15.78:0/1203812629 wait complete. 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 Processor -- start 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 -- start start 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feabc1006e0 0x7feabc10b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc1018d0 0x7feabc1096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feabc102290 0x7feabc109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feabc10d330 con 0x7feabc1018d0 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feabc10d1b0 con 0x7feabc102290 2024-03-16T11:04:05.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feac1631640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feabc10d4b0 con 0x7feabc1006e0 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feaba7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc1018d0 0x7feabc1096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feaba7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc1018d0 0x7feabc1096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36112/0 (socket says 172.21.15.78:36112) 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feaba7fc640 1 -- 172.21.15.78:0/1002513971 learned_addr learned my addr 172.21.15.78:0/1002513971 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feabaffd640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feabc1006e0 0x7feabc10b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.312+0000 7feabb7fe640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feabc102290 0x7feabc109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaba7fc640 1 -- 172.21.15.78:0/1002513971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feabc1006e0 msgr2=0x7feabc10b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaba7fc640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feabc1006e0 0x7feabc10b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaba7fc640 1 -- 172.21.15.78:0/1002513971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feabc102290 msgr2=0x7feabc109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaba7fc640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feabc102290 0x7feabc109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaba7fc640 1 -- 172.21.15.78:0/1002513971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feabc10a470 con 0x7feabc1018d0 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feabaffd640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feabc1006e0 0x7feabc10b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaba7fc640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc1018d0 0x7feabc1096a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feab000e9f0 tx=0x7feab000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaa3fff640 1 -- 172.21.15.78:0/1002513971 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab000cd80 con 0x7feabc1018d0 2024-03-16T11:04:05.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feabc1a1af0 con 0x7feabc1018d0 2024-03-16T11:04:05.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaa3fff640 1 -- 172.21.15.78:0/1002513971 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feab00108e0 con 0x7feabc1018d0 2024-03-16T11:04:05.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feaa3fff640 1 -- 172.21.15.78:0/1002513971 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0018920 con 0x7feabc1018d0 2024-03-16T11:04:05.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.313+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feabc1a2090 con 0x7feabc1018d0 2024-03-16T11:04:05.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.314+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feabc107a20 con 0x7feabc1018d0 2024-03-16T11:04:05.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.315+0000 7feaa3fff640 1 -- 172.21.15.78:0/1002513971 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7feab0010a90 con 0x7feabc1018d0 2024-03-16T11:04:05.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.316+0000 7feaa3fff640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7feaa40a2800 0x7feaa40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.316+0000 7feaa3fff640 1 -- 172.21.15.78:0/1002513971 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7feab00cf0a0 con 0x7feabc1018d0 2024-03-16T11:04:05.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.316+0000 7feabaffd640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7feaa40a2800 0x7feaa40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.316+0000 7feabaffd640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7feaa40a2800 0x7feaa40a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7feabc101730 tx=0x7feaac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:05.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.319+0000 7feaa3fff640 1 -- 172.21.15.78:0/1002513971 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feab009bc40 con 0x7feabc1018d0 2024-03-16T11:04:05.464 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:04:05 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: 2024-03-16T11:04:05.051+0000 7fa1f23ed700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-16T11:04:05.464 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:04:05 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: 2024-03-16T11:04:05.051+0000 7fa1f23ed700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-16T11:04:05.464 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:04:05 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: 2024-03-16T11:04:05.051+0000 7fa1f23ed700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-16T11:04:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.522+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7feabc061c20 con 0x7feaa40a2800 2024-03-16T11:04:05.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.525+0000 7feaa3fff640 1 -- 172.21.15.78:0/1002513971 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7feabc061c20 con 0x7feaa40a2800 2024-03-16T11:04:05.526 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_0/31b1212f-132d-4bb4-a342-539dd397529d 2024-03-16T11:04:05.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.527+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7feaa40a2800 msgr2=0x7feaa40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.527+0000 7feac1631640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7feaa40a2800 0x7feaa40a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7feabc101730 tx=0x7feaac009500 comp rx=0 tx=0).stop 2024-03-16T11:04:05.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.527+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc1018d0 msgr2=0x7feabc1096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.527+0000 7feac1631640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc1018d0 0x7feabc1096a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feab000e9f0 tx=0x7feab000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 shutdown_connections 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feabc102290 0x7feabc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc1018d0 0x7feabc1096a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7feaa40a2800 0x7feaa40a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 --2- 172.21.15.78:0/1002513971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feabc1006e0 0x7feabc10b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 >> 172.21.15.78:0/1002513971 conn(0x7feabc0fbc70 msgr2=0x7feabc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 shutdown_connections 2024-03-16T11:04:05.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.528+0000 7feac1631640 1 -- 172.21.15.78:0/1002513971 wait complete. 2024-03-16T11:04:05.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-16T11:04:05.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.644+0000 7fc538c7e640 1 -- 172.21.15.78:0/2752454295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 msgr2=0x7fc534109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.644+0000 7fc538c7e640 1 --2- 172.21.15.78:0/2752454295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc534109c40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009ca0 tx=0x7fc52c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- 172.21.15.78:0/2752454295 shutdown_connections 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 --2- 172.21.15.78:0/2752454295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc534109c40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 --2- 172.21.15.78:0/2752454295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5341029f0 0x7fc534102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 --2- 172.21.15.78:0/2752454295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 0x7fc534101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- 172.21.15.78:0/2752454295 >> 172.21.15.78:0/2752454295 conn(0x7fc5340fcd70 msgr2=0x7fc5340ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- 172.21.15.78:0/2752454295 shutdown_connections 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- 172.21.15.78:0/2752454295 wait complete. 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 Processor -- start 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- start start 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 0x7fc5341979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5341029f0 0x7fc534197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc53419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc53410d330 con 0x7fc5341033b0 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc53410d1b0 con 0x7fc534101800 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.645+0000 7fc538c7e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc53410d4b0 con 0x7fc5341029f0 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc53419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc53259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 0x7fc5341979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc53259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 0x7fc5341979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:57544/0 (socket says 172.21.15.78:57544) 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc53419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44362/0 (socket says 172.21.15.78:44362) 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc53259c640 1 -- 172.21.15.78:0/190953152 learned_addr learned my addr 172.21.15.78:0/190953152 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc52bfff640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5341029f0 0x7fc534197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 -- 172.21.15.78:0/190953152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5341029f0 msgr2=0x7fc534197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5341029f0 0x7fc534197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 -- 172.21.15.78:0/190953152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 msgr2=0x7fc5341979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 0x7fc5341979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 -- 172.21.15.78:0/190953152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53419f610 con 0x7fc5341033b0 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc53259c640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 0x7fc5341979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc532d9d640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc53419efa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc52c02c630 tx=0x7fc52c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc52b7fe640 1 -- 172.21.15.78:0/190953152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c041020 con 0x7fc5341033b0 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc52b7fe640 1 -- 172.21.15.78:0/190953152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc52c003e20 con 0x7fc5341033b0 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53419f8a0 con 0x7fc5341033b0 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc52b7fe640 1 -- 172.21.15.78:0/190953152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c039040 con 0x7fc5341033b0 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.646+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc53419fd90 con 0x7fc5341033b0 2024-03-16T11:04:05.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.647+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc534108bf0 con 0x7fc5341033b0 2024-03-16T11:04:05.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.648+0000 7fc52b7fe640 1 -- 172.21.15.78:0/190953152 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc52c044020 con 0x7fc5341033b0 2024-03-16T11:04:05.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.649+0000 7fc52b7fe640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5200a2800 0x7fc5200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:05.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.649+0000 7fc53259c640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5200a2800 0x7fc5200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:05.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.649+0000 7fc52b7fe640 1 -- 172.21.15.78:0/190953152 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fc52c0f0ac0 con 0x7fc5341033b0 2024-03-16T11:04:05.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.649+0000 7fc53259c640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5200a2800 0x7fc5200a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc5240095f0 tx=0x7fc524005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:05.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.653+0000 7fc52b7fe640 1 -- 172.21.15.78:0/190953152 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc52c04b020 con 0x7fc5341033b0 2024-03-16T11:04:05.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:05 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/218216407' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T11:04:05.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:05 smithi153 ceph-mon[35521]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:04:05.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:05 smithi153 ceph-mon[35521]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:05.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:05 smithi153 ceph-mon[35521]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:05.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:05 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/218216407' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T11:04:05.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:05 smithi078 ceph-mon[32235]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:04:05.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:05 smithi078 ceph-mon[32235]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:05.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:05 smithi078 ceph-mon[32235]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:05.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.855+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc534061cb0 con 0x7fc5200a2800 2024-03-16T11:04:05.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.878+0000 7fc52b7fe640 1 -- 172.21.15.78:0/190953152 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc534061cb0 con 0x7fc5200a2800 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5200a2800 msgr2=0x7fc5200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5200a2800 0x7fc5200a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc5240095f0 tx=0x7fc524005f90 comp rx=0 tx=0).stop 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 msgr2=0x7fc53419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc53419efa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc52c02c630 tx=0x7fc52c0328a0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 shutdown_connections 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5341033b0 0x7fc53419efa0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5200a2800 0x7fc5200a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5341029f0 0x7fc534197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 --2- 172.21.15.78:0/190953152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc534101800 0x7fc5341979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 >> 172.21.15.78:0/190953152 conn(0x7fc5340fcd70 msgr2=0x7fc534105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 shutdown_connections 2024-03-16T11:04:05.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.880+0000 7fc538c7e640 1 -- 172.21.15.78:0/190953152 wait complete. 2024-03-16T11:04:05.891 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-16T11:04:05.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:05 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/218216407' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T11:04:05.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:05 smithi098 ceph-mon[35848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:04:05.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:05 smithi098 ceph-mon[35848]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:05.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:05 smithi098 ceph-mon[35848]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 -- 172.21.15.78:0/1052458584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3074100fd0 msgr2=0x7f3074101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 --2- 172.21.15.78:0/1052458584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3074100fd0 0x7f3074101450 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3068009900 tx=0x7f306802bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 -- 172.21.15.78:0/1052458584 shutdown_connections 2024-03-16T11:04:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 --2- 172.21.15.78:0/1052458584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3074101990 0x7f30741082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 --2- 172.21.15.78:0/1052458584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3074100fd0 0x7f3074101450 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 --2- 172.21.15.78:0/1052458584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30740ffe80 0x7f30741002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 -- 172.21.15.78:0/1052458584 >> 172.21.15.78:0/1052458584 conn(0x7f30740fb410 msgr2=0x7f30740fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 -- 172.21.15.78:0/1052458584 shutdown_connections 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.997+0000 7f307a5a9640 1 -- 172.21.15.78:0/1052458584 wait complete. 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 Processor -- start 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 -- start start 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30740ffe80 0x7f3074197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3074100fd0 0x7f3074197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3074101990 0x7f307419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f307410d390 con 0x7f30740ffe80 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f307410d210 con 0x7f3074101990 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f307a5a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f307410d510 con 0x7f3074100fd0 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30740ffe80 0x7f3074197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30740ffe80 0x7f3074197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44386/0 (socket says 172.21.15.78:44386) 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 -- 172.21.15.78:0/2514727389 learned_addr learned my addr 172.21.15.78:0/2514727389 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3078b46640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3074101990 0x7f307419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f30737fe640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3074100fd0 0x7f3074197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 -- 172.21.15.78:0/2514727389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3074100fd0 msgr2=0x7f3074197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3074100fd0 0x7f3074197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 -- 172.21.15.78:0/2514727389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3074101990 msgr2=0x7f307419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3074101990 0x7f307419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.998+0000 7f3073fff640 1 -- 172.21.15.78:0/2514727389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307419f670 con 0x7f30740ffe80 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.999+0000 7f3073fff640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30740ffe80 0x7f3074197a50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f306400e9f0 tx=0x7f306400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.999+0000 7f30717fa640 1 -- 172.21.15.78:0/2514727389 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306400ccc0 con 0x7f30740ffe80 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.999+0000 7f30717fa640 1 -- 172.21.15.78:0/2514727389 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30640108e0 con 0x7f30740ffe80 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.999+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307419f960 con 0x7f30740ffe80 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.999+0000 7f30717fa640 1 -- 172.21.15.78:0/2514727389 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30640187c0 con 0x7f30740ffe80 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:05.999+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f307419fe80 con 0x7f30740ffe80 2024-03-16T11:04:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.000+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30741072b0 con 0x7f30740ffe80 2024-03-16T11:04:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.001+0000 7f30717fa640 1 -- 172.21.15.78:0/2514727389 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3064010a90 con 0x7f30740ffe80 2024-03-16T11:04:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.001+0000 7f30717fa640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f305c0a2800 0x7f305c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.001+0000 7f30737fe640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f305c0a2800 0x7f305c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.001+0000 7f30717fa640 1 -- 172.21.15.78:0/2514727389 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f30640cecf0 con 0x7f30740ffe80 2024-03-16T11:04:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.002+0000 7f30737fe640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f305c0a2800 0x7f305c0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3068005ab0 tx=0x7f3068005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.006+0000 7f30717fa640 1 -- 172.21.15.78:0/2514727389 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30640d3070 con 0x7f30740ffe80 2024-03-16T11:04:06.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.210+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f3074061cb0 con 0x7f305c0a2800 2024-03-16T11:04:06.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.213+0000 7f30717fa640 1 -- 172.21.15.78:0/2514727389 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3074061cb0 con 0x7f305c0a2800 2024-03-16T11:04:06.214 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_1/7fd86398-a306-4051-b5f6-32961c826f7e 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f305c0a2800 msgr2=0x7f305c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f305c0a2800 0x7f305c0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3068005ab0 tx=0x7f3068005fd0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30740ffe80 msgr2=0x7f3074197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30740ffe80 0x7f3074197a50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f306400e9f0 tx=0x7f306400edf0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 shutdown_connections 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3074101990 0x7f307419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f305c0a2800 0x7f305c0a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3074100fd0 0x7f3074197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 --2- 172.21.15.78:0/2514727389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30740ffe80 0x7f3074197a50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 >> 172.21.15.78:0/2514727389 conn(0x7f30740fb410 msgr2=0x7f3074104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:06.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 shutdown_connections 2024-03-16T11:04:06.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.215+0000 7f307a5a9640 1 -- 172.21.15.78:0/2514727389 wait complete. 2024-03-16T11:04:06.227 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3506025252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 msgr2=0x7f09c4101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3506025252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4101c10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009900 tx=0x7f09b402bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3506025252 shutdown_connections 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3506025252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c41033a0 0x7f09c4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3506025252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c41029e0 0x7f09c4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3506025252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4101c10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3506025252 >> 172.21.15.78:0/3506025252 conn(0x7f09c40fcdc0 msgr2=0x7f09c40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:06.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3506025252 shutdown_connections 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.328+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3506025252 wait complete. 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 Processor -- start 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 -- start start 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c41029e0 0x7f09c4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c41033a0 0x7f09c419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f09c410d320 con 0x7f09c41017f0 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09c410d1a0 con 0x7f09c41033a0 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c8ea1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09c410d4a0 con 0x7f09c41029e0 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44408/0 (socket says 172.21.15.78:44408) 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 -- 172.21.15.78:0/3314010773 learned_addr learned my addr 172.21.15.78:0/3314010773 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09b9d9b640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c41029e0 0x7f09c4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c2d9d640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c41033a0 0x7f09c419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 -- 172.21.15.78:0/3314010773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c41029e0 msgr2=0x7f09c4197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c41029e0 0x7f09c4197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 -- 172.21.15.78:0/3314010773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c41033a0 msgr2=0x7f09c419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c41033a0 0x7f09c419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 -- 172.21.15.78:0/3314010773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c419f630 con 0x7f09c41017f0 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09b9d9b640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c41029e0 0x7f09c4197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.329+0000 7f09c259c640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4197a10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f09b4000c00 tx=0x7f09b4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.330+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4043020 con 0x7f09c41017f0 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.330+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09b4004350 con 0x7f09c41017f0 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.330+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c419f8c0 con 0x7f09c41017f0 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.330+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4003b80 con 0x7f09c41017f0 2024-03-16T11:04:06.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.330+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c419fe10 con 0x7f09c41017f0 2024-03-16T11:04:06.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.331+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c4108be0 con 0x7f09c41017f0 2024-03-16T11:04:06.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.332+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f09b4046020 con 0x7f09c41017f0 2024-03-16T11:04:06.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.332+0000 7f09bbfff640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f09b00a2800 0x7f09b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.332+0000 7f09b9d9b640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f09b00a2800 0x7f09b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.332+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f09b40f0a30 con 0x7f09c41017f0 2024-03-16T11:04:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.333+0000 7f09b9d9b640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f09b00a2800 0x7f09b00a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f09c4198f20 tx=0x7f09ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.337+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f09b404d020 con 0x7f09c41017f0 2024-03-16T11:04:06.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.492+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f09b40b9b90 con 0x7f09c41017f0 2024-03-16T11:04:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.553+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09c402d1b0 con 0x7f09c41017f0 2024-03-16T11:04:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.554+0000 7f09bbfff640 1 -- 172.21.15.78:0/3314010773 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11869 (secure 0 0 0) 0x7f09b40bab70 con 0x7f09c41017f0 2024-03-16T11:04:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:06.555 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:04:03.658334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24488,24505,24521]},"id":1}]} 2024-03-16T11:04:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 21 2024-03-16T11:04:06.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f09b00a2800 msgr2=0x7f09b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f09b00a2800 0x7f09b00a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f09c4198f20 tx=0x7f09ac005f90 comp rx=0 tx=0).stop 2024-03-16T11:04:06.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 msgr2=0x7f09c4197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4197a10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f09b4000c00 tx=0x7f09b4005dc0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 shutdown_connections 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09c41033a0 0x7f09c419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f09b00a2800 0x7f09b00a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c41029e0 0x7f09c4197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 --2- 172.21.15.78:0/3314010773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09c41017f0 0x7f09c4197a10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.556+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 >> 172.21.15.78:0/3314010773 conn(0x7f09c40fcdc0 msgr2=0x7f09c4105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.557+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 shutdown_connections 2024-03-16T11:04:06.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.557+0000 7f09c8ea1640 1 -- 172.21.15.78:0/3314010773 wait complete. 2024-03-16T11:04:06.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/357509952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b80ff890 msgr2=0x7ff2b80ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/357509952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b80ff890 0x7ff2b80ffd10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009900 tx=0x7ff2a802bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/357509952 shutdown_connections 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/357509952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2b8100250 0x7ff2b8109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/357509952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b80ff890 0x7ff2b80ffd10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/357509952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2b8102e20 0x7ff2b8103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/357509952 >> 172.21.15.78:0/357509952 conn(0x7ff2b80fb410 msgr2=0x7ff2b80fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/357509952 shutdown_connections 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.669+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/357509952 wait complete. 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 Processor -- start 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 -- start start 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2b80ff890 0x7ff2b81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b8100250 0x7ff2b8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2b8102e20 0x7ff2b819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b810d330 con 0x7ff2b8100250 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b810d1b0 con 0x7ff2b80ff890 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bf4f1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b810d4b0 con 0x7ff2b8102e20 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bca8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b8100250 0x7ff2b8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bca8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b8100250 0x7ff2b8197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44422/0 (socket says 172.21.15.78:44422) 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bca8c640 1 -- 172.21.15.78:0/4107787558 learned_addr learned my addr 172.21.15.78:0/4107787558 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bd28d640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2b80ff890 0x7ff2b81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.670+0000 7ff2bda8e640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2b8102e20 0x7ff2b819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bca8c640 1 -- 172.21.15.78:0/4107787558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2b8102e20 msgr2=0x7ff2b819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bca8c640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2b8102e20 0x7ff2b819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bca8c640 1 -- 172.21.15.78:0/4107787558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2b80ff890 msgr2=0x7ff2b81979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bca8c640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2b80ff890 0x7ff2b81979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bca8c640 1 -- 172.21.15.78:0/4107787558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b819f610 con 0x7ff2b8100250 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bca8c640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b8100250 0x7ff2b8197f30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8006010 tx=0x7ff2a8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2ae7fc640 1 -- 172.21.15.78:0/4107787558 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8043020 con 0x7ff2b8100250 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2ae7fc640 1 -- 172.21.15.78:0/4107787558 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2a80043d0 con 0x7ff2b8100250 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b819f8a0 con 0x7ff2b8100250 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2ae7fc640 1 -- 172.21.15.78:0/4107787558 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8003b80 con 0x7ff2b8100250 2024-03-16T11:04:06.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.671+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b819fe40 con 0x7ff2b8100250 2024-03-16T11:04:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.672+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b8108ce0 con 0x7ff2b8100250 2024-03-16T11:04:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.673+0000 7ff2ae7fc640 1 -- 172.21.15.78:0/4107787558 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff2a8046020 con 0x7ff2b8100250 2024-03-16T11:04:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.673+0000 7ff2ae7fc640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2a40a2910 0x7ff2a40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.673+0000 7ff2bd28d640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2a40a2910 0x7ff2a40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.673+0000 7ff2ae7fc640 1 -- 172.21.15.78:0/4107787558 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff2a80f0aa0 con 0x7ff2b8100250 2024-03-16T11:04:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.674+0000 7ff2bd28d640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2a40a2910 0x7ff2a40a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8069b70 tx=0x7ff2b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.678+0000 7ff2ae7fc640 1 -- 172.21.15.78:0/4107787558 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2a804d020 con 0x7ff2b8100250 2024-03-16T11:04:06.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:06 smithi153 ceph-mon[35521]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:06.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:06 smithi153 ceph-mon[35521]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:06.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:06 smithi153 ceph-mon[35521]: pgmap v290: 97 pgs: 97 active+clean; 587 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5.4 KiB/s wr, 17 op/s 2024-03-16T11:04:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:06 smithi078 ceph-mon[32235]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:06 smithi078 ceph-mon[32235]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:06.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:06 smithi078 ceph-mon[32235]: pgmap v290: 97 pgs: 97 active+clean; 587 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5.4 KiB/s wr, 17 op/s 2024-03-16T11:04:06.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.853+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2b802d360 con 0x7ff2b8100250 2024-03-16T11:04:06.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.855+0000 7ff2ae7fc640 1 -- 172.21.15.78:0/4107787558 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22136 (secure 0 0 0) 0x7ff2a80bdca0 con 0x7ff2b8100250 2024-03-16T11:04:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:06.856 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":88,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:04:01.279417+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:04:06.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2a40a2910 msgr2=0x7ff2a40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2a40a2910 0x7ff2a40a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8069b70 tx=0x7ff2b4005f90 comp rx=0 tx=0).stop 2024-03-16T11:04:06.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b8100250 msgr2=0x7ff2b8197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b8100250 0x7ff2b8197f30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8006010 tx=0x7ff2a8005e40 comp rx=0 tx=0).stop 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 shutdown_connections 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff2a40a2910 0x7ff2a40a4dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2b8102e20 0x7ff2b819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b8100250 0x7ff2b8197f30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 --2- 172.21.15.78:0/4107787558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2b80ff890 0x7ff2b81979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 >> 172.21.15.78:0/4107787558 conn(0x7ff2b80fb410 msgr2=0x7ff2b8105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 shutdown_connections 2024-03-16T11:04:06.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.857+0000 7ff2bf4f1640 1 -- 172.21.15.78:0/4107787558 wait complete. 2024-03-16T11:04:06.869 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T11:04:06.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T11:04:06.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:06 smithi098 ceph-mon[35848]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:06.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:06 smithi098 ceph-mon[35848]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:06.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:06 smithi098 ceph-mon[35848]: pgmap v290: 97 pgs: 97 active+clean; 587 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5.4 KiB/s wr, 17 op/s 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 -- 172.21.15.78:0/1988310284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8288102340 msgr2=0x7f8288104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 --2- 172.21.15.78:0/1988310284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8288102340 0x7f8288104730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f827c009900 tx=0x7f827c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 -- 172.21.15.78:0/1988310284 shutdown_connections 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 --2- 172.21.15.78:0/1988310284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82881075a0 0x7f8288109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 --2- 172.21.15.78:0/1988310284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8288104c70 0x7f8288107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 --2- 172.21.15.78:0/1988310284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8288102340 0x7f8288104730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 -- 172.21.15.78:0/1988310284 >> 172.21.15.78:0/1988310284 conn(0x7f82880fbc90 msgr2=0x7f82880fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 -- 172.21.15.78:0/1988310284 shutdown_connections 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.970+0000 7f828ebc4640 1 -- 172.21.15.78:0/1988310284 wait complete. 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 Processor -- start 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 -- start start 2024-03-16T11:04:06.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8288102340 0x7f8288197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8288104c70 0x7f8288197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82881075a0 0x7f828819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f828810d360 con 0x7f82881075a0 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f828810d1e0 con 0x7f8288104c70 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f828810d4e0 con 0x7f8288102340 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82881075a0 0x7f828819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82881075a0 0x7f828819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44440/0 (socket says 172.21.15.78:44440) 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 -- 172.21.15.78:0/3859486046 learned_addr learned my addr 172.21.15.78:0/3859486046 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f8287fff640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8288104c70 0x7f8288197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828c960640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8288102340 0x7f8288197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 -- 172.21.15.78:0/3859486046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8288102340 msgr2=0x7f8288197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8288102340 0x7f8288197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 -- 172.21.15.78:0/3859486046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8288104c70 msgr2=0x7f8288197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:06.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8288104c70 0x7f8288197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:06.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 -- 172.21.15.78:0/3859486046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828819f6b0 con 0x7f82881075a0 2024-03-16T11:04:06.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828d161640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82881075a0 0x7f828819f040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f828000eb10 tx=0x7f828000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f8285ffb640 1 -- 172.21.15.78:0/3859486046 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280004b20 con 0x7f82881075a0 2024-03-16T11:04:06.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828819f9a0 con 0x7f82881075a0 2024-03-16T11:04:06.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f8285ffb640 1 -- 172.21.15.78:0/3859486046 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8280004cc0 con 0x7f82881075a0 2024-03-16T11:04:06.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.971+0000 7f8285ffb640 1 -- 172.21.15.78:0/3859486046 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280010690 con 0x7f82881075a0 2024-03-16T11:04:06.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.972+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f828819ff40 con 0x7f82881075a0 2024-03-16T11:04:06.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.973+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82880fd780 con 0x7f82881075a0 2024-03-16T11:04:06.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.974+0000 7f8285ffb640 1 -- 172.21.15.78:0/3859486046 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8280005020 con 0x7f82881075a0 2024-03-16T11:04:06.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.974+0000 7f8285ffb640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82780a2890 0x7f82780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:06.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.974+0000 7f828c960640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82780a2890 0x7f82780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:06.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.974+0000 7f8285ffb640 1 -- 172.21.15.78:0/3859486046 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f82800cf100 con 0x7f82881075a0 2024-03-16T11:04:06.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.975+0000 7f828c960640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82780a2890 0x7f82780a4d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f827c009230 tx=0x7f827c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:06.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:06.979+0000 7f8285ffb640 1 -- 172.21.15.78:0/3859486046 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f828009c1f0 con 0x7f82881075a0 2024-03-16T11:04:07.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.151+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8288061c20 con 0x7f82780a2890 2024-03-16T11:04:07.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.158+0000 7f8285ffb640 1 -- 172.21.15.78:0/3859486046 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258761 (secure 0 0 0) 0x7f8288061c20 con 0x7f82780a2890 2024-03-16T11:04:07.162 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:07.163 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82780a2890 msgr2=0x7f82780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82780a2890 0x7f82780a4d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f827c009230 tx=0x7f827c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82881075a0 msgr2=0x7f828819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82881075a0 0x7f828819f040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f828000eb10 tx=0x7f828000c2d0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 shutdown_connections 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82881075a0 0x7f828819f040 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8288104c70 0x7f8288197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82780a2890 0x7f82780a4d50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 --2- 172.21.15.78:0/3859486046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8288102340 0x7f8288197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 >> 172.21.15.78:0/3859486046 conn(0x7f82880fbc90 msgr2=0x7f8288102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:07.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.161+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 shutdown_connections 2024-03-16T11:04:07.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.162+0000 7f828ebc4640 1 -- 172.21.15.78:0/3859486046 wait complete. 2024-03-16T11:04:07.173 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-16T11:04:05.965828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601406,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":148,"num_read_kb":126,"num_write":273,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":197,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370512,"kb_used_data":12980,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571312,"statfs":{"total":1151940427776,"available":1151561023488,"internally_reserved":0,"allocated":13291520,"data_stored":7206194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11038,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.694559"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293052+0000","last_change":"2024-03-16T11:04:00.295528+0000","last_active":"2024-03-16T11:04:01.293052+0000","last_peered":"2024-03-16T11:04:01.293052+0000","last_clean":"2024-03-16T11:04:01.293052+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:04:01.293052+0000","last_undegraded":"2024-03-16T11:04:01.293052+0000","last_fullsized":"2024-03-16T11:04:01.293052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:22.852488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211243+0000","last_change":"2024-03-16T11:04:00.289043+0000","last_active":"2024-03-16T11:04:02.211243+0000","last_peered":"2024-03-16T11:04:02.211243+0000","last_clean":"2024-03-16T11:04:02.211243+0000","last_became_active":"2024-03-16T11:04:00.288742+0000","last_became_peered":"2024-03-16T11:04:00.288742+0000","last_unstale":"2024-03-16T11:04:02.211243+0000","last_undegraded":"2024-03-16T11:04:02.211243+0000","last_fullsized":"2024-03-16T11:04:02.211243+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:07:56.710499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211195+0000","last_change":"2024-03-16T11:04:00.293443+0000","last_active":"2024-03-16T11:04:02.211195+0000","last_peered":"2024-03-16T11:04:02.211195+0000","last_clean":"2024-03-16T11:04:02.211195+0000","last_became_active":"2024-03-16T11:04:00.293117+0000","last_became_peered":"2024-03-16T11:04:00.293117+0000","last_unstale":"2024-03-16T11:04:02.211195+0000","last_undegraded":"2024-03-16T11:04:02.211195+0000","last_fullsized":"2024-03-16T11:04:02.211195+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:49.728721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211345+0000","last_change":"2024-03-16T11:04:00.292105+0000","last_active":"2024-03-16T11:04:02.211345+0000","last_peered":"2024-03-16T11:04:02.211345+0000","last_clean":"2024-03-16T11:04:02.211345+0000","last_became_active":"2024-03-16T11:04:00.291755+0000","last_became_peered":"2024-03-16T11:04:00.291755+0000","last_unstale":"2024-03-16T11:04:02.211345+0000","last_undegraded":"2024-03-16T11:04:02.211345+0000","last_fullsized":"2024-03-16T11:04:02.211345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:48:43.734854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292996+0000","last_change":"2024-03-16T11:04:00.282168+0000","last_active":"2024-03-16T11:04:01.292996+0000","last_peered":"2024-03-16T11:04:01.292996+0000","last_clean":"2024-03-16T11:04:01.292996+0000","last_became_active":"2024-03-16T11:04:00.281753+0000","last_became_peered":"2024-03-16T11:04:00.281753+0000","last_unstale":"2024-03-16T11:04:01.292996+0000","last_undegraded":"2024-03-16T11:04:01.292996+0000","last_fullsized":"2024-03-16T11:04:01.292996+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:13.223955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288332+0000","last_change":"2024-03-16T11:04:00.292584+0000","last_active":"2024-03-16T11:04:01.288332+0000","last_peered":"2024-03-16T11:04:01.288332+0000","last_clean":"2024-03-16T11:04:01.288332+0000","last_became_active":"2024-03-16T11:04:00.292255+0000","last_became_peered":"2024-03-16T11:04:00.292255+0000","last_unstale":"2024-03-16T11:04:01.288332+0000","last_undegraded":"2024-03-16T11:04:01.288332+0000","last_fullsized":"2024-03-16T11:04:01.288332+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:38:08.926471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211533+0000","last_change":"2024-03-16T11:04:00.292446+0000","last_active":"2024-03-16T11:04:02.211533+0000","last_peered":"2024-03-16T11:04:02.211533+0000","last_clean":"2024-03-16T11:04:02.211533+0000","last_became_active":"2024-03-16T11:04:00.292079+0000","last_became_peered":"2024-03-16T11:04:00.292079+0000","last_unstale":"2024-03-16T11:04:02.211533+0000","last_undegraded":"2024-03-16T11:04:02.211533+0000","last_fullsized":"2024-03-16T11:04:02.211533+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.325945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292412+0000","last_change":"2024-03-16T11:04:00.294644+0000","last_active":"2024-03-16T11:04:01.292412+0000","last_peered":"2024-03-16T11:04:01.292412+0000","last_clean":"2024-03-16T11:04:01.292412+0000","last_became_active":"2024-03-16T11:04:00.294283+0000","last_became_peered":"2024-03-16T11:04:00.294283+0000","last_unstale":"2024-03-16T11:04:01.292412+0000","last_undegraded":"2024-03-16T11:04:01.292412+0000","last_fullsized":"2024-03-16T11:04:01.292412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:43.125823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210952+0000","last_change":"2024-03-16T11:04:00.302512+0000","last_active":"2024-03-16T11:04:02.210952+0000","last_peered":"2024-03-16T11:04:02.210952+0000","last_clean":"2024-03-16T11:04:02.210952+0000","last_became_active":"2024-03-16T11:04:00.301891+0000","last_became_peered":"2024-03-16T11:04:00.301891+0000","last_unstale":"2024-03-16T11:04:02.210952+0000","last_undegraded":"2024-03-16T11:04:02.210952+0000","last_fullsized":"2024-03-16T11:04:02.210952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:15.112295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288314+0000","last_change":"2024-03-16T11:04:00.293327+0000","last_active":"2024-03-16T11:04:01.288314+0000","last_peered":"2024-03-16T11:04:01.288314+0000","last_clean":"2024-03-16T11:04:01.288314+0000","last_became_active":"2024-03-16T11:04:00.292971+0000","last_became_peered":"2024-03-16T11:04:00.292971+0000","last_unstale":"2024-03-16T11:04:01.288314+0000","last_undegraded":"2024-03-16T11:04:01.288314+0000","last_fullsized":"2024-03-16T11:04:01.288314+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:48:48.565861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211624+0000","last_change":"2024-03-16T11:04:00.298956+0000","last_active":"2024-03-16T11:04:02.211624+0000","last_peered":"2024-03-16T11:04:02.211624+0000","last_clean":"2024-03-16T11:04:02.211624+0000","last_became_active":"2024-03-16T11:04:00.298508+0000","last_became_peered":"2024-03-16T11:04:00.298508+0000","last_unstale":"2024-03-16T11:04:02.211624+0000","last_undegraded":"2024-03-16T11:04:02.211624+0000","last_fullsized":"2024-03-16T11:04:02.211624+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:25:46.576514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211196+0000","last_change":"2024-03-16T11:04:00.294453+0000","last_active":"2024-03-16T11:04:02.211196+0000","last_peered":"2024-03-16T11:04:02.211196+0000","last_clean":"2024-03-16T11:04:02.211196+0000","last_became_active":"2024-03-16T11:04:00.294084+0000","last_became_peered":"2024-03-16T11:04:00.294084+0000","last_unstale":"2024-03-16T11:04:02.211196+0000","last_undegraded":"2024-03-16T11:04:02.211196+0000","last_fullsized":"2024-03-16T11:04:02.211196+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:12.924835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210917+0000","last_change":"2024-03-16T11:04:00.293149+0000","last_active":"2024-03-16T11:04:02.210917+0000","last_peered":"2024-03-16T11:04:02.210917+0000","last_clean":"2024-03-16T11:04:02.210917+0000","last_became_active":"2024-03-16T11:04:00.292839+0000","last_became_peered":"2024-03-16T11:04:00.292839+0000","last_unstale":"2024-03-16T11:04:02.210917+0000","last_undegraded":"2024-03-16T11:04:02.210917+0000","last_fullsized":"2024-03-16T11:04:02.210917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:34:38.446918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210764+0000","last_change":"2024-03-16T11:04:00.285748+0000","last_active":"2024-03-16T11:04:02.210764+0000","last_peered":"2024-03-16T11:04:02.210764+0000","last_clean":"2024-03-16T11:04:02.210764+0000","last_became_active":"2024-03-16T11:04:00.285394+0000","last_became_peered":"2024-03-16T11:04:00.285394+0000","last_unstale":"2024-03-16T11:04:02.210764+0000","last_undegraded":"2024-03-16T11:04:02.210764+0000","last_fullsized":"2024-03-16T11:04:02.210764+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:02.113271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211121+0000","last_change":"2024-03-16T11:04:00.301783+0000","last_active":"2024-03-16T11:04:02.211121+0000","last_peered":"2024-03-16T11:04:02.211121+0000","last_clean":"2024-03-16T11:04:02.211121+0000","last_became_active":"2024-03-16T11:04:00.301313+0000","last_became_peered":"2024-03-16T11:04:00.301313+0000","last_unstale":"2024-03-16T11:04:02.211121+0000","last_undegraded":"2024-03-16T11:04:02.211121+0000","last_fullsized":"2024-03-16T11:04:02.211121+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:19:32.243602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210351+0000","last_change":"2024-03-16T11:04:00.301375+0000","last_active":"2024-03-16T11:04:02.210351+0000","last_peered":"2024-03-16T11:04:02.210351+0000","last_clean":"2024-03-16T11:04:02.210351+0000","last_became_active":"2024-03-16T11:04:00.301013+0000","last_became_peered":"2024-03-16T11:04:00.301013+0000","last_unstale":"2024-03-16T11:04:02.210351+0000","last_undegraded":"2024-03-16T11:04:02.210351+0000","last_fullsized":"2024-03-16T11:04:02.210351+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:21:52.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293034+0000","last_change":"2024-03-16T11:04:00.296287+0000","last_active":"2024-03-16T11:04:01.293034+0000","last_peered":"2024-03-16T11:04:01.293034+0000","last_clean":"2024-03-16T11:04:01.293034+0000","last_became_active":"2024-03-16T11:04:00.295972+0000","last_became_peered":"2024-03-16T11:04:00.295972+0000","last_unstale":"2024-03-16T11:04:01.293034+0000","last_undegraded":"2024-03-16T11:04:01.293034+0000","last_fullsized":"2024-03-16T11:04:01.293034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:14.586052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.296337+0000","last_change":"2024-03-16T11:04:00.295551+0000","last_active":"2024-03-16T11:04:00.296337+0000","last_peered":"2024-03-16T11:04:00.296337+0000","last_clean":"2024-03-16T11:04:00.296337+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:04:00.296337+0000","last_undegraded":"2024-03-16T11:04:00.296337+0000","last_fullsized":"2024-03-16T11:04:00.296337+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:17.910862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.291493+0000","last_change":"2024-03-16T11:04:00.290455+0000","last_active":"2024-03-16T11:04:00.291493+0000","last_peered":"2024-03-16T11:04:00.291493+0000","last_clean":"2024-03-16T11:04:00.291493+0000","last_became_active":"2024-03-16T11:04:00.290057+0000","last_became_peered":"2024-03-16T11:04:00.290057+0000","last_unstale":"2024-03-16T11:04:00.291493+0000","last_undegraded":"2024-03-16T11:04:00.291493+0000","last_fullsized":"2024-03-16T11:04:00.291493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:33.368456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.304226+0000","last_change":"2024-03-16T11:03:59.285059+0000","last_active":"2024-03-16T11:04:00.304226+0000","last_peered":"2024-03-16T11:04:00.304226+0000","last_clean":"2024-03-16T11:04:00.304226+0000","last_became_active":"2024-03-16T11:03:59.284745+0000","last_became_peered":"2024-03-16T11:03:59.284745+0000","last_unstale":"2024-03-16T11:04:00.304226+0000","last_undegraded":"2024-03-16T11:04:00.304226+0000","last_fullsized":"2024-03-16T11:04:00.304226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:12.516958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292685+0000","last_change":"2024-03-16T11:04:00.292933+0000","last_active":"2024-03-16T11:04:01.292685+0000","last_peered":"2024-03-16T11:04:01.292685+0000","last_clean":"2024-03-16T11:04:01.292685+0000","last_became_active":"2024-03-16T11:04:00.292637+0000","last_became_peered":"2024-03-16T11:04:00.292637+0000","last_unstale":"2024-03-16T11:04:01.292685+0000","last_undegraded":"2024-03-16T11:04:01.292685+0000","last_fullsized":"2024-03-16T11:04:01.292685+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:22.231735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292837+0000","last_change":"2024-03-16T11:03:59.285267+0000","last_active":"2024-03-16T11:04:01.292837+0000","last_peered":"2024-03-16T11:04:01.292837+0000","last_clean":"2024-03-16T11:04:01.292837+0000","last_became_active":"2024-03-16T11:03:59.284890+0000","last_became_peered":"2024-03-16T11:03:59.284890+0000","last_unstale":"2024-03-16T11:04:01.292837+0000","last_undegraded":"2024-03-16T11:04:01.292837+0000","last_fullsized":"2024-03-16T11:04:01.292837+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:53.625612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211339+0000","last_change":"2024-03-16T11:04:00.292660+0000","last_active":"2024-03-16T11:04:02.211339+0000","last_peered":"2024-03-16T11:04:02.211339+0000","last_clean":"2024-03-16T11:04:02.211339+0000","last_became_active":"2024-03-16T11:04:00.292366+0000","last_became_peered":"2024-03-16T11:04:00.292366+0000","last_unstale":"2024-03-16T11:04:02.211339+0000","last_undegraded":"2024-03-16T11:04:02.211339+0000","last_fullsized":"2024-03-16T11:04:02.211339+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:36:44.331456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293550+0000","last_change":"2024-03-16T11:03:59.286370+0000","last_active":"2024-03-16T11:04:01.293550+0000","last_peered":"2024-03-16T11:04:01.293550+0000","last_clean":"2024-03-16T11:04:01.293550+0000","last_became_active":"2024-03-16T11:03:59.286065+0000","last_became_peered":"2024-03-16T11:03:59.286065+0000","last_unstale":"2024-03-16T11:04:01.293550+0000","last_undegraded":"2024-03-16T11:04:01.293550+0000","last_fullsized":"2024-03-16T11:04:01.293550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:20.126848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210412+0000","last_change":"2024-03-16T11:04:00.290864+0000","last_active":"2024-03-16T11:04:02.210412+0000","last_peered":"2024-03-16T11:04:02.210412+0000","last_clean":"2024-03-16T11:04:02.210412+0000","last_became_active":"2024-03-16T11:04:00.290537+0000","last_became_peered":"2024-03-16T11:04:00.290537+0000","last_unstale":"2024-03-16T11:04:02.210412+0000","last_undegraded":"2024-03-16T11:04:02.210412+0000","last_fullsized":"2024-03-16T11:04:02.210412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:52.991767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293423+0000","last_change":"2024-03-16T11:03:59.281663+0000","last_active":"2024-03-16T11:04:01.293423+0000","last_peered":"2024-03-16T11:04:01.293423+0000","last_clean":"2024-03-16T11:04:01.293423+0000","last_became_active":"2024-03-16T11:03:59.281328+0000","last_became_peered":"2024-03-16T11:03:59.281328+0000","last_unstale":"2024-03-16T11:04:01.293423+0000","last_undegraded":"2024-03-16T11:04:01.293423+0000","last_fullsized":"2024-03-16T11:04:01.293423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:44.511109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010964+0000","last_change":"2024-03-16T11:04:00.297607+0000","last_active":"2024-03-16T11:04:02.010964+0000","last_peered":"2024-03-16T11:04:02.010964+0000","last_clean":"2024-03-16T11:04:02.010964+0000","last_became_active":"2024-03-16T11:04:00.297014+0000","last_became_peered":"2024-03-16T11:04:00.297014+0000","last_unstale":"2024-03-16T11:04:02.010964+0000","last_undegraded":"2024-03-16T11:04:02.010964+0000","last_fullsized":"2024-03-16T11:04:02.010964+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:07:33.726437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.472340+0000","last_change":"2024-03-16T11:03:59.286300+0000","last_active":"2024-03-16T11:04:03.472340+0000","last_peered":"2024-03-16T11:04:03.472340+0000","last_clean":"2024-03-16T11:04:03.472340+0000","last_became_active":"2024-03-16T11:03:59.285764+0000","last_became_peered":"2024-03-16T11:03:59.285764+0000","last_unstale":"2024-03-16T11:04:03.472340+0000","last_undegraded":"2024-03-16T11:04:03.472340+0000","last_fullsized":"2024-03-16T11:04:03.472340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:49.084245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293106+0000","last_change":"2024-03-16T11:04:00.289152+0000","last_active":"2024-03-16T11:04:01.293106+0000","last_peered":"2024-03-16T11:04:01.293106+0000","last_clean":"2024-03-16T11:04:01.293106+0000","last_became_active":"2024-03-16T11:04:00.288837+0000","last_became_peered":"2024-03-16T11:04:00.288837+0000","last_unstale":"2024-03-16T11:04:01.293106+0000","last_undegraded":"2024-03-16T11:04:01.293106+0000","last_fullsized":"2024-03-16T11:04:01.293106+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:45:01.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.312605+0000","last_change":"2024-03-16T11:03:59.281505+0000","last_active":"2024-03-16T11:04:03.312605+0000","last_peered":"2024-03-16T11:04:03.312605+0000","last_clean":"2024-03-16T11:04:03.312605+0000","last_became_active":"2024-03-16T11:03:59.281175+0000","last_became_peered":"2024-03-16T11:03:59.281175+0000","last_unstale":"2024-03-16T11:04:03.312605+0000","last_undegraded":"2024-03-16T11:04:03.312605+0000","last_fullsized":"2024-03-16T11:04:03.312605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:12.463914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010775+0000","last_change":"2024-03-16T11:04:00.297769+0000","last_active":"2024-03-16T11:04:02.010775+0000","last_peered":"2024-03-16T11:04:02.010775+0000","last_clean":"2024-03-16T11:04:02.010775+0000","last_became_active":"2024-03-16T11:04:00.297061+0000","last_became_peered":"2024-03-16T11:04:00.297061+0000","last_unstale":"2024-03-16T11:04:02.010775+0000","last_undegraded":"2024-03-16T11:04:02.010775+0000","last_fullsized":"2024-03-16T11:04:02.010775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:42.594315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.314859+0000","last_change":"2024-03-16T11:03:59.290260+0000","last_active":"2024-03-16T11:04:03.314859+0000","last_peered":"2024-03-16T11:04:03.314859+0000","last_clean":"2024-03-16T11:04:03.314859+0000","last_became_active":"2024-03-16T11:03:59.289843+0000","last_became_peered":"2024-03-16T11:03:59.289843+0000","last_unstale":"2024-03-16T11:04:03.314859+0000","last_undegraded":"2024-03-16T11:04:03.314859+0000","last_fullsized":"2024-03-16T11:04:03.314859+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:22.483488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292645+0000","last_change":"2024-03-16T11:04:00.287873+0000","last_active":"2024-03-16T11:04:01.292645+0000","last_peered":"2024-03-16T11:04:01.292645+0000","last_clean":"2024-03-16T11:04:01.292645+0000","last_became_active":"2024-03-16T11:04:00.287002+0000","last_became_peered":"2024-03-16T11:04:00.287002+0000","last_unstale":"2024-03-16T11:04:01.292645+0000","last_undegraded":"2024-03-16T11:04:01.292645+0000","last_fullsized":"2024-03-16T11:04:01.292645+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:57.851726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010567+0000","last_change":"2024-03-16T11:03:59.284745+0000","last_active":"2024-03-16T11:04:02.010567+0000","last_peered":"2024-03-16T11:04:02.010567+0000","last_clean":"2024-03-16T11:04:02.010567+0000","last_became_active":"2024-03-16T11:03:59.284325+0000","last_became_peered":"2024-03-16T11:03:59.284325+0000","last_unstale":"2024-03-16T11:04:02.010567+0000","last_undegraded":"2024-03-16T11:04:02.010567+0000","last_fullsized":"2024-03-16T11:04:02.010567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.537787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.291120+0000","last_change":"2024-03-16T11:04:00.290698+0000","last_active":"2024-03-16T11:04:00.291120+0000","last_peered":"2024-03-16T11:04:00.291120+0000","last_clean":"2024-03-16T11:04:00.291120+0000","last_became_active":"2024-03-16T11:04:00.290348+0000","last_became_peered":"2024-03-16T11:04:00.290348+0000","last_unstale":"2024-03-16T11:04:00.291120+0000","last_undegraded":"2024-03-16T11:04:00.291120+0000","last_fullsized":"2024-03-16T11:04:00.291120+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:49:02.626376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.470716+0000","last_change":"2024-03-16T11:03:59.280705+0000","last_active":"2024-03-16T11:04:03.470716+0000","last_peered":"2024-03-16T11:04:03.470716+0000","last_clean":"2024-03-16T11:04:03.470716+0000","last_became_active":"2024-03-16T11:03:59.280372+0000","last_became_peered":"2024-03-16T11:03:59.280372+0000","last_unstale":"2024-03-16T11:04:03.470716+0000","last_undegraded":"2024-03-16T11:04:03.470716+0000","last_fullsized":"2024-03-16T11:04:03.470716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:54.187322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210790+0000","last_change":"2024-03-16T11:04:00.293867+0000","last_active":"2024-03-16T11:04:02.210790+0000","last_peered":"2024-03-16T11:04:02.210790+0000","last_clean":"2024-03-16T11:04:02.210790+0000","last_became_active":"2024-03-16T11:04:00.293330+0000","last_became_peered":"2024-03-16T11:04:00.293330+0000","last_unstale":"2024-03-16T11:04:02.210790+0000","last_undegraded":"2024-03-16T11:04:02.210790+0000","last_fullsized":"2024-03-16T11:04:02.210790+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:54.798346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310441+0000","last_change":"2024-03-16T11:03:59.284202+0000","last_active":"2024-03-16T11:04:02.310441+0000","last_peered":"2024-03-16T11:04:02.310441+0000","last_clean":"2024-03-16T11:04:02.310441+0000","last_became_active":"2024-03-16T11:03:59.283656+0000","last_became_peered":"2024-03-16T11:03:59.283656+0000","last_unstale":"2024-03-16T11:04:02.310441+0000","last_undegraded":"2024-03-16T11:04:02.310441+0000","last_fullsized":"2024-03-16T11:04:02.310441+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:44.760307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211278+0000","last_change":"2024-03-16T11:04:00.301753+0000","last_active":"2024-03-16T11:04:02.211278+0000","last_peered":"2024-03-16T11:04:02.211278+0000","last_clean":"2024-03-16T11:04:02.211278+0000","last_became_active":"2024-03-16T11:04:00.301425+0000","last_became_peered":"2024-03-16T11:04:00.301425+0000","last_unstale":"2024-03-16T11:04:02.211278+0000","last_undegraded":"2024-03-16T11:04:02.211278+0000","last_fullsized":"2024-03-16T11:04:02.211278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:30.889952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309722+0000","last_change":"2024-03-16T11:03:59.288268+0000","last_active":"2024-03-16T11:04:02.309722+0000","last_peered":"2024-03-16T11:04:02.309722+0000","last_clean":"2024-03-16T11:04:02.309722+0000","last_became_active":"2024-03-16T11:03:59.287861+0000","last_became_peered":"2024-03-16T11:03:59.287861+0000","last_unstale":"2024-03-16T11:04:02.309722+0000","last_undegraded":"2024-03-16T11:04:02.309722+0000","last_fullsized":"2024-03-16T11:04:02.309722+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:34:34.248408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210678+0000","last_change":"2024-03-16T11:04:00.306490+0000","last_active":"2024-03-16T11:04:02.210678+0000","last_peered":"2024-03-16T11:04:02.210678+0000","last_clean":"2024-03-16T11:04:02.210678+0000","last_became_active":"2024-03-16T11:04:00.305874+0000","last_became_peered":"2024-03-16T11:04:00.305874+0000","last_unstale":"2024-03-16T11:04:02.210678+0000","last_undegraded":"2024-03-16T11:04:02.210678+0000","last_fullsized":"2024-03-16T11:04:02.210678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'11","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:04.830366+0000","last_change":"2024-03-16T11:03:59.280714+0000","last_active":"2024-03-16T11:04:04.830366+0000","last_peered":"2024-03-16T11:04:04.830366+0000","last_clean":"2024-03-16T11:04:04.830366+0000","last_became_active":"2024-03-16T11:03:59.280398+0000","last_became_peered":"2024-03-16T11:03:59.280398+0000","last_unstale":"2024-03-16T11:04:04.830366+0000","last_undegraded":"2024-03-16T11:04:04.830366+0000","last_fullsized":"2024-03-16T11:04:04.830366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:34.256125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293439+0000","last_change":"2024-03-16T11:04:00.285669+0000","last_active":"2024-03-16T11:04:01.293439+0000","last_peered":"2024-03-16T11:04:01.293439+0000","last_clean":"2024-03-16T11:04:01.293439+0000","last_became_active":"2024-03-16T11:04:00.285337+0000","last_became_peered":"2024-03-16T11:04:00.285337+0000","last_unstale":"2024-03-16T11:04:01.293439+0000","last_undegraded":"2024-03-16T11:04:01.293439+0000","last_fullsized":"2024-03-16T11:04:01.293439+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:36.925350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.306876+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:04:00.306876+0000","last_peered":"2024-03-16T11:04:00.306876+0000","last_clean":"2024-03-16T11:04:00.306876+0000","last_became_active":"2024-03-16T11:03:59.289357+0000","last_became_peered":"2024-03-16T11:03:59.289357+0000","last_unstale":"2024-03-16T11:04:00.306876+0000","last_undegraded":"2024-03-16T11:04:00.306876+0000","last_fullsized":"2024-03-16T11:04:00.306876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:26:35.860091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210766+0000","last_change":"2024-03-16T11:04:00.298488+0000","last_active":"2024-03-16T11:04:02.210766+0000","last_peered":"2024-03-16T11:04:02.210766+0000","last_clean":"2024-03-16T11:04:02.210766+0000","last_became_active":"2024-03-16T11:04:00.298202+0000","last_became_peered":"2024-03-16T11:04:00.298202+0000","last_unstale":"2024-03-16T11:04:02.210766+0000","last_undegraded":"2024-03-16T11:04:02.210766+0000","last_fullsized":"2024-03-16T11:04:02.210766+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:48.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310110+0000","last_change":"2024-03-16T11:03:59.287066+0000","last_active":"2024-03-16T11:04:02.310110+0000","last_peered":"2024-03-16T11:04:02.310110+0000","last_clean":"2024-03-16T11:04:02.310110+0000","last_became_active":"2024-03-16T11:03:59.286731+0000","last_became_peered":"2024-03-16T11:03:59.286731+0000","last_unstale":"2024-03-16T11:04:02.310110+0000","last_undegraded":"2024-03-16T11:04:02.310110+0000","last_fullsized":"2024-03-16T11:04:02.310110+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:51.443833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211723+0000","last_change":"2024-03-16T11:04:00.294450+0000","last_active":"2024-03-16T11:04:02.211723+0000","last_peered":"2024-03-16T11:04:02.211723+0000","last_clean":"2024-03-16T11:04:02.211723+0000","last_became_active":"2024-03-16T11:04:00.293769+0000","last_became_peered":"2024-03-16T11:04:00.293769+0000","last_unstale":"2024-03-16T11:04:02.211723+0000","last_undegraded":"2024-03-16T11:04:02.211723+0000","last_fullsized":"2024-03-16T11:04:02.211723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:02.755907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.472065+0000","last_change":"2024-03-16T11:03:59.286197+0000","last_active":"2024-03-16T11:04:03.472065+0000","last_peered":"2024-03-16T11:04:03.472065+0000","last_clean":"2024-03-16T11:04:03.472065+0000","last_became_active":"2024-03-16T11:03:59.285757+0000","last_became_peered":"2024-03-16T11:03:59.285757+0000","last_unstale":"2024-03-16T11:04:03.472065+0000","last_undegraded":"2024-03-16T11:04:03.472065+0000","last_fullsized":"2024-03-16T11:04:03.472065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:47.868106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211359+0000","last_change":"2024-03-16T11:04:00.296709+0000","last_active":"2024-03-16T11:04:02.211359+0000","last_peered":"2024-03-16T11:04:02.211359+0000","last_clean":"2024-03-16T11:04:02.211359+0000","last_became_active":"2024-03-16T11:04:00.296372+0000","last_became_peered":"2024-03-16T11:04:00.296372+0000","last_unstale":"2024-03-16T11:04:02.211359+0000","last_undegraded":"2024-03-16T11:04:02.211359+0000","last_fullsized":"2024-03-16T11:04:02.211359+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:37.948866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'10","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310921+0000","last_change":"2024-03-16T11:03:59.283831+0000","last_active":"2024-03-16T11:04:02.310921+0000","last_peered":"2024-03-16T11:04:02.310921+0000","last_clean":"2024-03-16T11:04:02.310921+0000","last_became_active":"2024-03-16T11:03:59.283404+0000","last_became_peered":"2024-03-16T11:03:59.283404+0000","last_unstale":"2024-03-16T11:04:02.310921+0000","last_undegraded":"2024-03-16T11:04:02.310921+0000","last_fullsized":"2024-03-16T11:04:02.310921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:26:34.012597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010572+0000","last_change":"2024-03-16T11:04:00.299703+0000","last_active":"2024-03-16T11:04:02.010572+0000","last_peered":"2024-03-16T11:04:02.010572+0000","last_clean":"2024-03-16T11:04:02.010572+0000","last_became_active":"2024-03-16T11:04:00.299321+0000","last_became_peered":"2024-03-16T11:04:00.299321+0000","last_unstale":"2024-03-16T11:04:02.010572+0000","last_undegraded":"2024-03-16T11:04:02.010572+0000","last_fullsized":"2024-03-16T11:04:02.010572+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.529281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293373+0000","last_change":"2024-03-16T11:03:59.277837+0000","last_active":"2024-03-16T11:04:01.293373+0000","last_peered":"2024-03-16T11:04:01.293373+0000","last_clean":"2024-03-16T11:04:01.293373+0000","last_became_active":"2024-03-16T11:03:59.277515+0000","last_became_peered":"2024-03-16T11:03:59.277515+0000","last_unstale":"2024-03-16T11:04:01.293373+0000","last_undegraded":"2024-03-16T11:04:01.293373+0000","last_fullsized":"2024-03-16T11:04:01.293373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:05.116990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292720+0000","last_change":"2024-03-16T11:04:00.292998+0000","last_active":"2024-03-16T11:04:01.292720+0000","last_peered":"2024-03-16T11:04:01.292720+0000","last_clean":"2024-03-16T11:04:01.292720+0000","last_became_active":"2024-03-16T11:04:00.292608+0000","last_became_peered":"2024-03-16T11:04:00.292608+0000","last_unstale":"2024-03-16T11:04:01.292720+0000","last_undegraded":"2024-03-16T11:04:01.292720+0000","last_fullsized":"2024-03-16T11:04:01.292720+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:41.996826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.276099+0000","last_change":"2024-03-16T11:03:59.283896+0000","last_active":"2024-03-16T11:04:00.276099+0000","last_peered":"2024-03-16T11:04:00.276099+0000","last_clean":"2024-03-16T11:04:00.276099+0000","last_became_active":"2024-03-16T11:03:59.283562+0000","last_became_peered":"2024-03-16T11:03:59.283562+0000","last_unstale":"2024-03-16T11:04:00.276099+0000","last_undegraded":"2024-03-16T11:04:00.276099+0000","last_fullsized":"2024-03-16T11:04:00.276099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:15:02.466629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288131+0000","last_change":"2024-03-16T11:04:00.288897+0000","last_active":"2024-03-16T11:04:01.288131+0000","last_peered":"2024-03-16T11:04:01.288131+0000","last_clean":"2024-03-16T11:04:01.288131+0000","last_became_active":"2024-03-16T11:04:00.288493+0000","last_became_peered":"2024-03-16T11:04:00.288493+0000","last_unstale":"2024-03-16T11:04:01.288131+0000","last_undegraded":"2024-03-16T11:04:01.288131+0000","last_fullsized":"2024-03-16T11:04:01.288131+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:31.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211778+0000","last_change":"2024-03-16T11:03:59.291595+0000","last_active":"2024-03-16T11:04:02.211778+0000","last_peered":"2024-03-16T11:04:02.211778+0000","last_clean":"2024-03-16T11:04:02.211778+0000","last_became_active":"2024-03-16T11:03:59.291159+0000","last_became_peered":"2024-03-16T11:03:59.291159+0000","last_unstale":"2024-03-16T11:04:02.211778+0000","last_undegraded":"2024-03-16T11:04:02.211778+0000","last_fullsized":"2024-03-16T11:04:02.211778+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:03.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288103+0000","last_change":"2024-03-16T11:04:00.302356+0000","last_active":"2024-03-16T11:04:01.288103+0000","last_peered":"2024-03-16T11:04:01.288103+0000","last_clean":"2024-03-16T11:04:01.288103+0000","last_became_active":"2024-03-16T11:04:00.302025+0000","last_became_peered":"2024-03-16T11:04:00.302025+0000","last_unstale":"2024-03-16T11:04:01.288103+0000","last_undegraded":"2024-03-16T11:04:01.288103+0000","last_fullsized":"2024-03-16T11:04:01.288103+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:46.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010722+0000","last_change":"2024-03-16T11:04:00.298400+0000","last_active":"2024-03-16T11:04:02.010722+0000","last_peered":"2024-03-16T11:04:02.010722+0000","last_clean":"2024-03-16T11:04:02.010722+0000","last_became_active":"2024-03-16T11:04:00.298026+0000","last_became_peered":"2024-03-16T11:04:00.298026+0000","last_unstale":"2024-03-16T11:04:02.010722+0000","last_undegraded":"2024-03-16T11:04:02.010722+0000","last_fullsized":"2024-03-16T11:04:02.010722+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:08.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.313933+0000","last_change":"2024-03-16T11:03:59.287528+0000","last_active":"2024-03-16T11:04:03.313933+0000","last_peered":"2024-03-16T11:04:03.313933+0000","last_clean":"2024-03-16T11:04:03.313933+0000","last_became_active":"2024-03-16T11:03:59.287218+0000","last_became_peered":"2024-03-16T11:03:59.287218+0000","last_unstale":"2024-03-16T11:04:03.313933+0000","last_undegraded":"2024-03-16T11:04:03.313933+0000","last_fullsized":"2024-03-16T11:04:03.313933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:19.803233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293020+0000","last_change":"2024-03-16T11:04:00.292857+0000","last_active":"2024-03-16T11:04:01.293020+0000","last_peered":"2024-03-16T11:04:01.293020+0000","last_clean":"2024-03-16T11:04:01.293020+0000","last_became_active":"2024-03-16T11:04:00.292574+0000","last_became_peered":"2024-03-16T11:04:00.292574+0000","last_unstale":"2024-03-16T11:04:01.293020+0000","last_undegraded":"2024-03-16T11:04:01.293020+0000","last_fullsized":"2024-03-16T11:04:01.293020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:48.466780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.469229+0000","last_change":"2024-03-16T11:03:59.281251+0000","last_active":"2024-03-16T11:04:03.469229+0000","last_peered":"2024-03-16T11:04:03.469229+0000","last_clean":"2024-03-16T11:04:03.469229+0000","last_became_active":"2024-03-16T11:03:59.280830+0000","last_became_peered":"2024-03-16T11:03:59.280830+0000","last_unstale":"2024-03-16T11:04:03.469229+0000","last_undegraded":"2024-03-16T11:04:03.469229+0000","last_fullsized":"2024-03-16T11:04:03.469229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:03:04.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":342,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211284+0000","last_change":"2024-03-16T11:02:48.497749+0000","last_active":"2024-03-16T11:04:02.211284+0000","last_peered":"2024-03-16T11:04:02.211284+0000","last_clean":"2024-03-16T11:04:02.211284+0000","last_became_active":"2024-03-16T11:02:48.497157+0000","last_became_peered":"2024-03-16T11:02:48.497157+0000","last_unstale":"2024-03-16T11:04:02.211284+0000","last_undegraded":"2024-03-16T11:04:02.211284+0000","last_fullsized":"2024-03-16T11:04:02.211284+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_clean_scrub_stamp":"2024-03-16T10:59:26.309639+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:04.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210680+0000","last_change":"2024-03-16T11:04:00.300821+0000","last_active":"2024-03-16T11:04:02.210680+0000","last_peered":"2024-03-16T11:04:02.210680+0000","last_clean":"2024-03-16T11:04:02.210680+0000","last_became_active":"2024-03-16T11:04:00.300510+0000","last_became_peered":"2024-03-16T11:04:00.300510+0000","last_unstale":"2024-03-16T11:04:02.210680+0000","last_undegraded":"2024-03-16T11:04:02.210680+0000","last_fullsized":"2024-03-16T11:04:02.210680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:57.035167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309011+0000","last_change":"2024-03-16T11:03:59.283609+0000","last_active":"2024-03-16T11:04:02.309011+0000","last_peered":"2024-03-16T11:04:02.309011+0000","last_clean":"2024-03-16T11:04:02.309011+0000","last_became_active":"2024-03-16T11:03:59.283227+0000","last_became_peered":"2024-03-16T11:03:59.283227+0000","last_unstale":"2024-03-16T11:04:02.309011+0000","last_undegraded":"2024-03-16T11:04:02.309011+0000","last_fullsized":"2024-03-16T11:04:02.309011+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:18.904202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309087+0000","last_change":"2024-03-16T11:03:59.283485+0000","last_active":"2024-03-16T11:04:02.309087+0000","last_peered":"2024-03-16T11:04:02.309087+0000","last_clean":"2024-03-16T11:04:02.309087+0000","last_became_active":"2024-03-16T11:03:59.283145+0000","last_became_peered":"2024-03-16T11:03:59.283145+0000","last_unstale":"2024-03-16T11:04:02.309087+0000","last_undegraded":"2024-03-16T11:04:02.309087+0000","last_fullsized":"2024-03-16T11:04:02.309087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:51:53.143512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210844+0000","last_change":"2024-03-16T11:04:00.285134+0000","last_active":"2024-03-16T11:04:02.210844+0000","last_peered":"2024-03-16T11:04:02.210844+0000","last_clean":"2024-03-16T11:04:02.210844+0000","last_became_active":"2024-03-16T11:04:00.284705+0000","last_became_peered":"2024-03-16T11:04:00.284705+0000","last_unstale":"2024-03-16T11:04:02.210844+0000","last_undegraded":"2024-03-16T11:04:02.210844+0000","last_fullsized":"2024-03-16T11:04:02.210844+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:38.637970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.276221+0000","last_change":"2024-03-16T11:03:59.289999+0000","last_active":"2024-03-16T11:04:00.276221+0000","last_peered":"2024-03-16T11:04:00.276221+0000","last_clean":"2024-03-16T11:04:00.276221+0000","last_became_active":"2024-03-16T11:03:59.289694+0000","last_became_peered":"2024-03-16T11:03:59.289694+0000","last_unstale":"2024-03-16T11:04:00.276221+0000","last_undegraded":"2024-03-16T11:04:00.276221+0000","last_fullsized":"2024-03-16T11:04:00.276221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:16.037387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292886+0000","last_change":"2024-03-16T11:04:00.298148+0000","last_active":"2024-03-16T11:04:01.292886+0000","last_peered":"2024-03-16T11:04:01.292886+0000","last_clean":"2024-03-16T11:04:01.292886+0000","last_became_active":"2024-03-16T11:04:00.297822+0000","last_became_peered":"2024-03-16T11:04:00.297822+0000","last_unstale":"2024-03-16T11:04:01.292886+0000","last_undegraded":"2024-03-16T11:04:01.292886+0000","last_fullsized":"2024-03-16T11:04:01.292886+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:09:28.645317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.308275+0000","last_change":"2024-03-16T11:03:59.292207+0000","last_active":"2024-03-16T11:04:02.308275+0000","last_peered":"2024-03-16T11:04:02.308275+0000","last_clean":"2024-03-16T11:04:02.308275+0000","last_became_active":"2024-03-16T11:03:59.291859+0000","last_became_peered":"2024-03-16T11:03:59.291859+0000","last_unstale":"2024-03-16T11:04:02.308275+0000","last_undegraded":"2024-03-16T11:04:02.308275+0000","last_fullsized":"2024-03-16T11:04:02.308275+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:55.365720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211254+0000","last_change":"2024-03-16T11:04:00.291261+0000","last_active":"2024-03-16T11:04:02.211254+0000","last_peered":"2024-03-16T11:04:02.211254+0000","last_clean":"2024-03-16T11:04:02.211254+0000","last_became_active":"2024-03-16T11:04:00.290832+0000","last_became_peered":"2024-03-16T11:04:00.290832+0000","last_unstale":"2024-03-16T11:04:02.211254+0000","last_undegraded":"2024-03-16T11:04:02.211254+0000","last_fullsized":"2024-03-16T11:04:02.211254+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:10.418808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310908+0000","last_change":"2024-03-16T11:03:59.288014+0000","last_active":"2024-03-16T11:04:02.310908+0000","last_peered":"2024-03-16T11:04:02.310908+0000","last_clean":"2024-03-16T11:04:02.310908+0000","last_became_active":"2024-03-16T11:03:59.287679+0000","last_became_peered":"2024-03-16T11:03:59.287679+0000","last_unstale":"2024-03-16T11:04:02.310908+0000","last_undegraded":"2024-03-16T11:04:02.310908+0000","last_fullsized":"2024-03-16T11:04:02.310908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:16:39.156224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292913+0000","last_change":"2024-03-16T11:04:00.288047+0000","last_active":"2024-03-16T11:04:01.292913+0000","last_peered":"2024-03-16T11:04:01.292913+0000","last_clean":"2024-03-16T11:04:01.292913+0000","last_became_active":"2024-03-16T11:04:00.287619+0000","last_became_peered":"2024-03-16T11:04:00.287619+0000","last_unstale":"2024-03-16T11:04:01.292913+0000","last_undegraded":"2024-03-16T11:04:01.292913+0000","last_fullsized":"2024-03-16T11:04:01.292913+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:43.608349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292418+0000","last_change":"2024-03-16T11:03:59.288439+0000","last_active":"2024-03-16T11:04:01.292418+0000","last_peered":"2024-03-16T11:04:01.292418+0000","last_clean":"2024-03-16T11:04:01.292418+0000","last_became_active":"2024-03-16T11:03:59.288122+0000","last_became_peered":"2024-03-16T11:03:59.288122+0000","last_unstale":"2024-03-16T11:04:01.292418+0000","last_undegraded":"2024-03-16T11:04:01.292418+0000","last_fullsized":"2024-03-16T11:04:01.292418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:50:25.499608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292300+0000","last_change":"2024-03-16T11:04:00.290522+0000","last_active":"2024-03-16T11:04:01.292300+0000","last_peered":"2024-03-16T11:04:01.292300+0000","last_clean":"2024-03-16T11:04:01.292300+0000","last_became_active":"2024-03-16T11:04:00.290175+0000","last_became_peered":"2024-03-16T11:04:00.290175+0000","last_unstale":"2024-03-16T11:04:01.292300+0000","last_undegraded":"2024-03-16T11:04:01.292300+0000","last_fullsized":"2024-03-16T11:04:01.292300+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:34.113278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.279781+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:04:00.279781+0000","last_peered":"2024-03-16T11:04:00.279781+0000","last_clean":"2024-03-16T11:04:00.279781+0000","last_became_active":"2024-03-16T11:03:59.289367+0000","last_became_peered":"2024-03-16T11:03:59.289367+0000","last_unstale":"2024-03-16T11:04:00.279781+0000","last_undegraded":"2024-03-16T11:04:00.279781+0000","last_fullsized":"2024-03-16T11:04:00.279781+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:42:48.627116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292760+0000","last_change":"2024-03-16T11:04:00.287685+0000","last_active":"2024-03-16T11:04:01.292760+0000","last_peered":"2024-03-16T11:04:01.292760+0000","last_clean":"2024-03-16T11:04:01.292760+0000","last_became_active":"2024-03-16T11:04:00.287367+0000","last_became_peered":"2024-03-16T11:04:00.287367+0000","last_unstale":"2024-03-16T11:04:01.292760+0000","last_undegraded":"2024-03-16T11:04:01.292760+0000","last_fullsized":"2024-03-16T11:04:01.292760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:25:52.741651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309713+0000","last_change":"2024-03-16T11:03:59.286293+0000","last_active":"2024-03-16T11:04:02.309713+0000","last_peered":"2024-03-16T11:04:02.309713+0000","last_clean":"2024-03-16T11:04:02.309713+0000","last_became_active":"2024-03-16T11:03:59.285817+0000","last_became_peered":"2024-03-16T11:03:59.285817+0000","last_unstale":"2024-03-16T11:04:02.309713+0000","last_undegraded":"2024-03-16T11:04:02.309713+0000","last_fullsized":"2024-03-16T11:04:02.309713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:38:12.093762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211897+0000","last_change":"2024-03-16T11:04:00.299391+0000","last_active":"2024-03-16T11:04:02.211897+0000","last_peered":"2024-03-16T11:04:02.211897+0000","last_clean":"2024-03-16T11:04:02.211897+0000","last_became_active":"2024-03-16T11:04:00.298931+0000","last_became_peered":"2024-03-16T11:04:00.298931+0000","last_unstale":"2024-03-16T11:04:02.211897+0000","last_undegraded":"2024-03-16T11:04:02.211897+0000","last_fullsized":"2024-03-16T11:04:02.211897+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:12:11.900154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.280431+0000","last_change":"2024-03-16T11:03:59.292319+0000","last_active":"2024-03-16T11:04:00.280431+0000","last_peered":"2024-03-16T11:04:00.280431+0000","last_clean":"2024-03-16T11:04:00.280431+0000","last_became_active":"2024-03-16T11:03:59.291953+0000","last_became_peered":"2024-03-16T11:03:59.291953+0000","last_unstale":"2024-03-16T11:04:00.280431+0000","last_undegraded":"2024-03-16T11:04:00.280431+0000","last_fullsized":"2024-03-16T11:04:00.280431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:41:04.615726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211686+0000","last_change":"2024-03-16T11:04:00.296544+0000","last_active":"2024-03-16T11:04:02.211686+0000","last_peered":"2024-03-16T11:04:02.211686+0000","last_clean":"2024-03-16T11:04:02.211686+0000","last_became_active":"2024-03-16T11:04:00.296078+0000","last_became_peered":"2024-03-16T11:04:00.296078+0000","last_unstale":"2024-03-16T11:04:02.211686+0000","last_undegraded":"2024-03-16T11:04:02.211686+0000","last_fullsized":"2024-03-16T11:04:02.211686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:56:22.846734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010914+0000","last_change":"2024-03-16T11:03:59.287172+0000","last_active":"2024-03-16T11:04:02.010914+0000","last_peered":"2024-03-16T11:04:02.010914+0000","last_clean":"2024-03-16T11:04:02.010914+0000","last_became_active":"2024-03-16T11:03:59.286785+0000","last_became_peered":"2024-03-16T11:03:59.286785+0000","last_unstale":"2024-03-16T11:04:02.010914+0000","last_undegraded":"2024-03-16T11:04:02.010914+0000","last_fullsized":"2024-03-16T11:04:02.010914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:17:55.584618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211458+0000","last_change":"2024-03-16T11:04:00.296019+0000","last_active":"2024-03-16T11:04:02.211458+0000","last_peered":"2024-03-16T11:04:02.211458+0000","last_clean":"2024-03-16T11:04:02.211458+0000","last_became_active":"2024-03-16T11:04:00.295685+0000","last_became_peered":"2024-03-16T11:04:00.295685+0000","last_unstale":"2024-03-16T11:04:02.211458+0000","last_undegraded":"2024-03-16T11:04:02.211458+0000","last_fullsized":"2024-03-16T11:04:02.211458+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:43.406124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210585+0000","last_change":"2024-03-16T11:03:59.288822+0000","last_active":"2024-03-16T11:04:02.210585+0000","last_peered":"2024-03-16T11:04:02.210585+0000","last_clean":"2024-03-16T11:04:02.210585+0000","last_became_active":"2024-03-16T11:03:59.288516+0000","last_became_peered":"2024-03-16T11:03:59.288516+0000","last_unstale":"2024-03-16T11:04:02.210585+0000","last_undegraded":"2024-03-16T11:04:02.210585+0000","last_fullsized":"2024-03-16T11:04:02.210585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:32:38.862476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211853+0000","last_change":"2024-03-16T11:04:00.296117+0000","last_active":"2024-03-16T11:04:02.211853+0000","last_peered":"2024-03-16T11:04:02.211853+0000","last_clean":"2024-03-16T11:04:02.211853+0000","last_became_active":"2024-03-16T11:04:00.295664+0000","last_became_peered":"2024-03-16T11:04:00.295664+0000","last_unstale":"2024-03-16T11:04:02.211853+0000","last_undegraded":"2024-03-16T11:04:02.211853+0000","last_fullsized":"2024-03-16T11:04:02.211853+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.448616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292441+0000","last_change":"2024-03-16T11:04:00.295035+0000","last_active":"2024-03-16T11:04:01.292441+0000","last_peered":"2024-03-16T11:04:01.292441+0000","last_clean":"2024-03-16T11:04:01.292441+0000","last_became_active":"2024-03-16T11:04:00.294689+0000","last_became_peered":"2024-03-16T11:04:00.294689+0000","last_unstale":"2024-03-16T11:04:01.292441+0000","last_undegraded":"2024-03-16T11:04:01.292441+0000","last_fullsized":"2024-03-16T11:04:01.292441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:55.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.294047+0000","last_change":"2024-03-16T11:04:00.292831+0000","last_active":"2024-03-16T11:04:00.294047+0000","last_peered":"2024-03-16T11:04:00.294047+0000","last_clean":"2024-03-16T11:04:00.294047+0000","last_became_active":"2024-03-16T11:04:00.292387+0000","last_became_peered":"2024-03-16T11:04:00.292387+0000","last_unstale":"2024-03-16T11:04:00.294047+0000","last_undegraded":"2024-03-16T11:04:00.294047+0000","last_fullsized":"2024-03-16T11:04:00.294047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:13:38.248585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.293620+0000","last_change":"2024-03-16T11:04:00.291267+0000","last_active":"2024-03-16T11:04:00.293620+0000","last_peered":"2024-03-16T11:04:00.293620+0000","last_clean":"2024-03-16T11:04:00.293620+0000","last_became_active":"2024-03-16T11:04:00.290911+0000","last_became_peered":"2024-03-16T11:04:00.290911+0000","last_unstale":"2024-03-16T11:04:00.293620+0000","last_undegraded":"2024-03-16T11:04:00.293620+0000","last_fullsized":"2024-03-16T11:04:00.293620+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:11.909490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211280+0000","last_change":"2024-03-16T11:04:00.300470+0000","last_active":"2024-03-16T11:04:02.211280+0000","last_peered":"2024-03-16T11:04:02.211280+0000","last_clean":"2024-03-16T11:04:02.211280+0000","last_became_active":"2024-03-16T11:04:00.300107+0000","last_became_peered":"2024-03-16T11:04:00.300107+0000","last_unstale":"2024-03-16T11:04:02.211280+0000","last_undegraded":"2024-03-16T11:04:02.211280+0000","last_fullsized":"2024-03-16T11:04:02.211280+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:18.718749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210714+0000","last_change":"2024-03-16T11:04:00.300857+0000","last_active":"2024-03-16T11:04:02.210714+0000","last_peered":"2024-03-16T11:04:02.210714+0000","last_clean":"2024-03-16T11:04:02.210714+0000","last_became_active":"2024-03-16T11:04:00.300551+0000","last_became_peered":"2024-03-16T11:04:00.300551+0000","last_unstale":"2024-03-16T11:04:02.210714+0000","last_undegraded":"2024-03-16T11:04:02.210714+0000","last_fullsized":"2024-03-16T11:04:02.210714+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:24.362561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292613+0000","last_change":"2024-03-16T11:04:00.294060+0000","last_active":"2024-03-16T11:04:01.292613+0000","last_peered":"2024-03-16T11:04:01.292613+0000","last_clean":"2024-03-16T11:04:01.292613+0000","last_became_active":"2024-03-16T11:04:00.293615+0000","last_became_peered":"2024-03-16T11:04:00.293615+0000","last_unstale":"2024-03-16T11:04:01.292613+0000","last_undegraded":"2024-03-16T11:04:01.292613+0000","last_fullsized":"2024-03-16T11:04:01.292613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:28:33.789436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.297122+0000","last_change":"2024-03-16T11:04:00.294943+0000","last_active":"2024-03-16T11:04:00.297122+0000","last_peered":"2024-03-16T11:04:00.297122+0000","last_clean":"2024-03-16T11:04:00.297122+0000","last_became_active":"2024-03-16T11:04:00.294637+0000","last_became_peered":"2024-03-16T11:04:00.294637+0000","last_unstale":"2024-03-16T11:04:00.297122+0000","last_undegraded":"2024-03-16T11:04:00.297122+0000","last_fullsized":"2024-03-16T11:04:00.297122+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:37.630443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292728+0000","last_change":"2024-03-16T11:04:00.300737+0000","last_active":"2024-03-16T11:04:01.292728+0000","last_peered":"2024-03-16T11:04:01.292728+0000","last_clean":"2024-03-16T11:04:01.292728+0000","last_became_active":"2024-03-16T11:04:00.300410+0000","last_became_peered":"2024-03-16T11:04:00.300410+0000","last_unstale":"2024-03-16T11:04:01.292728+0000","last_undegraded":"2024-03-16T11:04:01.292728+0000","last_fullsized":"2024-03-16T11:04:01.292728+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:18.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010710+0000","last_change":"2024-03-16T11:04:00.292958+0000","last_active":"2024-03-16T11:04:02.010710+0000","last_peered":"2024-03-16T11:04:02.010710+0000","last_clean":"2024-03-16T11:04:02.010710+0000","last_became_active":"2024-03-16T11:04:00.292599+0000","last_became_peered":"2024-03-16T11:04:00.292599+0000","last_unstale":"2024-03-16T11:04:02.010710+0000","last_undegraded":"2024-03-16T11:04:02.010710+0000","last_fullsized":"2024-03-16T11:04:02.010710+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:46.370597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292521+0000","last_change":"2024-03-16T11:04:00.291381+0000","last_active":"2024-03-16T11:04:01.292521+0000","last_peered":"2024-03-16T11:04:01.292521+0000","last_clean":"2024-03-16T11:04:01.292521+0000","last_became_active":"2024-03-16T11:04:00.291046+0000","last_became_peered":"2024-03-16T11:04:00.291046+0000","last_unstale":"2024-03-16T11:04:01.292521+0000","last_undegraded":"2024-03-16T11:04:01.292521+0000","last_fullsized":"2024-03-16T11:04:01.292521+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:08.185733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210591+0000","last_change":"2024-03-16T11:04:00.301381+0000","last_active":"2024-03-16T11:04:02.210591+0000","last_peered":"2024-03-16T11:04:02.210591+0000","last_clean":"2024-03-16T11:04:02.210591+0000","last_became_active":"2024-03-16T11:04:00.300990+0000","last_became_peered":"2024-03-16T11:04:00.300990+0000","last_unstale":"2024-03-16T11:04:02.210591+0000","last_undegraded":"2024-03-16T11:04:02.210591+0000","last_fullsized":"2024-03-16T11:04:02.210591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:07:43.775066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210938+0000","last_change":"2024-03-16T11:04:00.298220+0000","last_active":"2024-03-16T11:04:02.210938+0000","last_peered":"2024-03-16T11:04:02.210938+0000","last_clean":"2024-03-16T11:04:02.210938+0000","last_became_active":"2024-03-16T11:04:00.297776+0000","last_became_peered":"2024-03-16T11:04:00.297776+0000","last_unstale":"2024-03-16T11:04:02.210938+0000","last_undegraded":"2024-03-16T11:04:02.210938+0000","last_fullsized":"2024-03-16T11:04:02.210938+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:40.387309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292369+0000","last_change":"2024-03-16T11:04:00.292988+0000","last_active":"2024-03-16T11:04:01.292369+0000","last_peered":"2024-03-16T11:04:01.292369+0000","last_clean":"2024-03-16T11:04:01.292369+0000","last_became_active":"2024-03-16T11:04:00.292655+0000","last_became_peered":"2024-03-16T11:04:00.292655+0000","last_unstale":"2024-03-16T11:04:01.292369+0000","last_undegraded":"2024-03-16T11:04:01.292369+0000","last_fullsized":"2024-03-16T11:04:01.292369+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:33:43.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":11038,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":20225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285581,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":453300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547219,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1585152,"data_stored":1046858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":937984,"data_stored":445312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005084,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":453904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266721,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1552384,"data_stored":1044872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430247,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":453390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":958464,"data_stored":454912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":454778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":446376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":954368,"data_stored":453404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888111,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":453944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1568768,"data_stored":1045144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":10181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:04:07.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T11:04:07.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 -- 172.21.15.78:0/2278711412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 msgr2=0x7f1a34103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:07.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 --2- 172.21.15.78:0/2278711412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 0x7f1a34103230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009900 tx=0x7f1a2402bc30 comp rx=0 tx=0).stop 2024-03-16T11:04:07.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 -- 172.21.15.78:0/2278711412 shutdown_connections 2024-03-16T11:04:07.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 --2- 172.21.15.78:0/2278711412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a34100240 0x7f1a34109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 --2- 172.21.15.78:0/2278711412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a340ff880 0x7f1a340ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 --2- 172.21.15.78:0/2278711412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 0x7f1a34103230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 -- 172.21.15.78:0/2278711412 >> 172.21.15.78:0/2278711412 conn(0x7f1a340fb420 msgr2=0x7f1a340fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 -- 172.21.15.78:0/2278711412 shutdown_connections 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 -- 172.21.15.78:0/2278711412 wait complete. 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.318+0000 7f1a39337640 1 Processor -- start 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a39337640 1 -- start start 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a39337640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a340ff880 0x7f1a34199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a39337640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a34100240 0x7f1a3419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a39337640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 0x7f1a341a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a39337640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3410d360 con 0x7f1a34102e10 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a39337640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3410d1e0 con 0x7f1a34100240 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a39337640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3410d4e0 con 0x7f1a340ff880 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a34100240 0x7f1a3419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:07.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a337fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 0x7f1a341a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a32ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a340ff880 0x7f1a34199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a34100240 0x7f1a3419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:57624/0 (socket says 172.21.15.78:57624) 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 -- 172.21.15.78:0/734655374 learned_addr learned my addr 172.21.15.78:0/734655374 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 -- 172.21.15.78:0/734655374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a340ff880 msgr2=0x7f1a34199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a340ff880 0x7f1a34199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 -- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 msgr2=0x7f1a341a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 0x7f1a341a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 -- 172.21.15.78:0/734655374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a341a18f0 con 0x7f1a34100240 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.319+0000 7f1a327fc640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a34100240 0x7f1a3419a1f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800e9f0 tx=0x7f1a2800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.320+0000 7f1a1bfff640 1 -- 172.21.15.78:0/734655374 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2800cd80 con 0x7f1a34100240 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.320+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a341a1be0 con 0x7f1a34100240 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.320+0000 7f1a1bfff640 1 -- 172.21.15.78:0/734655374 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a280108e0 con 0x7f1a34100240 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.320+0000 7f1a1bfff640 1 -- 172.21.15.78:0/734655374 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28018920 con 0x7f1a34100240 2024-03-16T11:04:07.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.320+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a341a2130 con 0x7f1a34100240 2024-03-16T11:04:07.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.321+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a341a2300 con 0x7f1a34100240 2024-03-16T11:04:07.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.322+0000 7f1a1bfff640 1 -- 172.21.15.78:0/734655374 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1a28010a90 con 0x7f1a34100240 2024-03-16T11:04:07.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.322+0000 7f1a1bfff640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1a1c0a2890 0x7f1a1c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:07.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.322+0000 7f1a1bfff640 1 -- 172.21.15.78:0/734655374 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f1a280cee60 con 0x7f1a34100240 2024-03-16T11:04:07.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.322+0000 7f1a32ffd640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1a1c0a2890 0x7f1a1c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:07.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.323+0000 7f1a32ffd640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1a1c0a2890 0x7f1a1c0a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009230 tx=0x7f1a24005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:07.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.327+0000 7f1a1bfff640 1 -- 172.21.15.78:0/734655374 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a2809bed0 con 0x7f1a34100240 2024-03-16T11:04:07.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.507+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a341020b0 con 0x7f1a1c0a2890 2024-03-16T11:04:07.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.521+0000 7f1a1bfff640 1 -- 172.21.15.78:0/734655374 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258761 (secure 0 0 0) 0x7f1a341020b0 con 0x7f1a1c0a2890 2024-03-16T11:04:07.524 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:07.525 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-16T11:04:07.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1a1c0a2890 msgr2=0x7f1a1c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:07.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1a1c0a2890 0x7f1a1c0a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009230 tx=0x7f1a24005be0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a34100240 msgr2=0x7f1a3419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a34100240 0x7f1a3419a1f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800e9f0 tx=0x7f1a2800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 shutdown_connections 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a34102e10 0x7f1a341a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a34100240 0x7f1a3419a1f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1a1c0a2890 0x7f1a1c0a4d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 --2- 172.21.15.78:0/734655374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a340ff880 0x7f1a34199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 >> 172.21.15.78:0/734655374 conn(0x7f1a340fb420 msgr2=0x7f1a34100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 shutdown_connections 2024-03-16T11:04:07.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:07.524+0000 7f1a39337640 1 -- 172.21.15.78:0/734655374 wait complete. 2024-03-16T11:04:07.536 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-16T11:04:05.965828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601406,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":148,"num_read_kb":126,"num_write":273,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":197,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370512,"kb_used_data":12980,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571312,"statfs":{"total":1151940427776,"available":1151561023488,"internally_reserved":0,"allocated":13291520,"data_stored":7206194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11038,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.694559"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293052+0000","last_change":"2024-03-16T11:04:00.295528+0000","last_active":"2024-03-16T11:04:01.293052+0000","last_peered":"2024-03-16T11:04:01.293052+0000","last_clean":"2024-03-16T11:04:01.293052+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:04:01.293052+0000","last_undegraded":"2024-03-16T11:04:01.293052+0000","last_fullsized":"2024-03-16T11:04:01.293052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:22.852488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211243+0000","last_change":"2024-03-16T11:04:00.289043+0000","last_active":"2024-03-16T11:04:02.211243+0000","last_peered":"2024-03-16T11:04:02.211243+0000","last_clean":"2024-03-16T11:04:02.211243+0000","last_became_active":"2024-03-16T11:04:00.288742+0000","last_became_peered":"2024-03-16T11:04:00.288742+0000","last_unstale":"2024-03-16T11:04:02.211243+0000","last_undegraded":"2024-03-16T11:04:02.211243+0000","last_fullsized":"2024-03-16T11:04:02.211243+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:07:56.710499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211195+0000","last_change":"2024-03-16T11:04:00.293443+0000","last_active":"2024-03-16T11:04:02.211195+0000","last_peered":"2024-03-16T11:04:02.211195+0000","last_clean":"2024-03-16T11:04:02.211195+0000","last_became_active":"2024-03-16T11:04:00.293117+0000","last_became_peered":"2024-03-16T11:04:00.293117+0000","last_unstale":"2024-03-16T11:04:02.211195+0000","last_undegraded":"2024-03-16T11:04:02.211195+0000","last_fullsized":"2024-03-16T11:04:02.211195+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:49.728721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211345+0000","last_change":"2024-03-16T11:04:00.292105+0000","last_active":"2024-03-16T11:04:02.211345+0000","last_peered":"2024-03-16T11:04:02.211345+0000","last_clean":"2024-03-16T11:04:02.211345+0000","last_became_active":"2024-03-16T11:04:00.291755+0000","last_became_peered":"2024-03-16T11:04:00.291755+0000","last_unstale":"2024-03-16T11:04:02.211345+0000","last_undegraded":"2024-03-16T11:04:02.211345+0000","last_fullsized":"2024-03-16T11:04:02.211345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:48:43.734854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292996+0000","last_change":"2024-03-16T11:04:00.282168+0000","last_active":"2024-03-16T11:04:01.292996+0000","last_peered":"2024-03-16T11:04:01.292996+0000","last_clean":"2024-03-16T11:04:01.292996+0000","last_became_active":"2024-03-16T11:04:00.281753+0000","last_became_peered":"2024-03-16T11:04:00.281753+0000","last_unstale":"2024-03-16T11:04:01.292996+0000","last_undegraded":"2024-03-16T11:04:01.292996+0000","last_fullsized":"2024-03-16T11:04:01.292996+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:13.223955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288332+0000","last_change":"2024-03-16T11:04:00.292584+0000","last_active":"2024-03-16T11:04:01.288332+0000","last_peered":"2024-03-16T11:04:01.288332+0000","last_clean":"2024-03-16T11:04:01.288332+0000","last_became_active":"2024-03-16T11:04:00.292255+0000","last_became_peered":"2024-03-16T11:04:00.292255+0000","last_unstale":"2024-03-16T11:04:01.288332+0000","last_undegraded":"2024-03-16T11:04:01.288332+0000","last_fullsized":"2024-03-16T11:04:01.288332+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:38:08.926471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211533+0000","last_change":"2024-03-16T11:04:00.292446+0000","last_active":"2024-03-16T11:04:02.211533+0000","last_peered":"2024-03-16T11:04:02.211533+0000","last_clean":"2024-03-16T11:04:02.211533+0000","last_became_active":"2024-03-16T11:04:00.292079+0000","last_became_peered":"2024-03-16T11:04:00.292079+0000","last_unstale":"2024-03-16T11:04:02.211533+0000","last_undegraded":"2024-03-16T11:04:02.211533+0000","last_fullsized":"2024-03-16T11:04:02.211533+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.325945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292412+0000","last_change":"2024-03-16T11:04:00.294644+0000","last_active":"2024-03-16T11:04:01.292412+0000","last_peered":"2024-03-16T11:04:01.292412+0000","last_clean":"2024-03-16T11:04:01.292412+0000","last_became_active":"2024-03-16T11:04:00.294283+0000","last_became_peered":"2024-03-16T11:04:00.294283+0000","last_unstale":"2024-03-16T11:04:01.292412+0000","last_undegraded":"2024-03-16T11:04:01.292412+0000","last_fullsized":"2024-03-16T11:04:01.292412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:43.125823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210952+0000","last_change":"2024-03-16T11:04:00.302512+0000","last_active":"2024-03-16T11:04:02.210952+0000","last_peered":"2024-03-16T11:04:02.210952+0000","last_clean":"2024-03-16T11:04:02.210952+0000","last_became_active":"2024-03-16T11:04:00.301891+0000","last_became_peered":"2024-03-16T11:04:00.301891+0000","last_unstale":"2024-03-16T11:04:02.210952+0000","last_undegraded":"2024-03-16T11:04:02.210952+0000","last_fullsized":"2024-03-16T11:04:02.210952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:15.112295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288314+0000","last_change":"2024-03-16T11:04:00.293327+0000","last_active":"2024-03-16T11:04:01.288314+0000","last_peered":"2024-03-16T11:04:01.288314+0000","last_clean":"2024-03-16T11:04:01.288314+0000","last_became_active":"2024-03-16T11:04:00.292971+0000","last_became_peered":"2024-03-16T11:04:00.292971+0000","last_unstale":"2024-03-16T11:04:01.288314+0000","last_undegraded":"2024-03-16T11:04:01.288314+0000","last_fullsized":"2024-03-16T11:04:01.288314+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:48:48.565861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211624+0000","last_change":"2024-03-16T11:04:00.298956+0000","last_active":"2024-03-16T11:04:02.211624+0000","last_peered":"2024-03-16T11:04:02.211624+0000","last_clean":"2024-03-16T11:04:02.211624+0000","last_became_active":"2024-03-16T11:04:00.298508+0000","last_became_peered":"2024-03-16T11:04:00.298508+0000","last_unstale":"2024-03-16T11:04:02.211624+0000","last_undegraded":"2024-03-16T11:04:02.211624+0000","last_fullsized":"2024-03-16T11:04:02.211624+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:25:46.576514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211196+0000","last_change":"2024-03-16T11:04:00.294453+0000","last_active":"2024-03-16T11:04:02.211196+0000","last_peered":"2024-03-16T11:04:02.211196+0000","last_clean":"2024-03-16T11:04:02.211196+0000","last_became_active":"2024-03-16T11:04:00.294084+0000","last_became_peered":"2024-03-16T11:04:00.294084+0000","last_unstale":"2024-03-16T11:04:02.211196+0000","last_undegraded":"2024-03-16T11:04:02.211196+0000","last_fullsized":"2024-03-16T11:04:02.211196+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:12.924835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210917+0000","last_change":"2024-03-16T11:04:00.293149+0000","last_active":"2024-03-16T11:04:02.210917+0000","last_peered":"2024-03-16T11:04:02.210917+0000","last_clean":"2024-03-16T11:04:02.210917+0000","last_became_active":"2024-03-16T11:04:00.292839+0000","last_became_peered":"2024-03-16T11:04:00.292839+0000","last_unstale":"2024-03-16T11:04:02.210917+0000","last_undegraded":"2024-03-16T11:04:02.210917+0000","last_fullsized":"2024-03-16T11:04:02.210917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:34:38.446918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210764+0000","last_change":"2024-03-16T11:04:00.285748+0000","last_active":"2024-03-16T11:04:02.210764+0000","last_peered":"2024-03-16T11:04:02.210764+0000","last_clean":"2024-03-16T11:04:02.210764+0000","last_became_active":"2024-03-16T11:04:00.285394+0000","last_became_peered":"2024-03-16T11:04:00.285394+0000","last_unstale":"2024-03-16T11:04:02.210764+0000","last_undegraded":"2024-03-16T11:04:02.210764+0000","last_fullsized":"2024-03-16T11:04:02.210764+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:02.113271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211121+0000","last_change":"2024-03-16T11:04:00.301783+0000","last_active":"2024-03-16T11:04:02.211121+0000","last_peered":"2024-03-16T11:04:02.211121+0000","last_clean":"2024-03-16T11:04:02.211121+0000","last_became_active":"2024-03-16T11:04:00.301313+0000","last_became_peered":"2024-03-16T11:04:00.301313+0000","last_unstale":"2024-03-16T11:04:02.211121+0000","last_undegraded":"2024-03-16T11:04:02.211121+0000","last_fullsized":"2024-03-16T11:04:02.211121+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:19:32.243602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210351+0000","last_change":"2024-03-16T11:04:00.301375+0000","last_active":"2024-03-16T11:04:02.210351+0000","last_peered":"2024-03-16T11:04:02.210351+0000","last_clean":"2024-03-16T11:04:02.210351+0000","last_became_active":"2024-03-16T11:04:00.301013+0000","last_became_peered":"2024-03-16T11:04:00.301013+0000","last_unstale":"2024-03-16T11:04:02.210351+0000","last_undegraded":"2024-03-16T11:04:02.210351+0000","last_fullsized":"2024-03-16T11:04:02.210351+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:21:52.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293034+0000","last_change":"2024-03-16T11:04:00.296287+0000","last_active":"2024-03-16T11:04:01.293034+0000","last_peered":"2024-03-16T11:04:01.293034+0000","last_clean":"2024-03-16T11:04:01.293034+0000","last_became_active":"2024-03-16T11:04:00.295972+0000","last_became_peered":"2024-03-16T11:04:00.295972+0000","last_unstale":"2024-03-16T11:04:01.293034+0000","last_undegraded":"2024-03-16T11:04:01.293034+0000","last_fullsized":"2024-03-16T11:04:01.293034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:14.586052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.296337+0000","last_change":"2024-03-16T11:04:00.295551+0000","last_active":"2024-03-16T11:04:00.296337+0000","last_peered":"2024-03-16T11:04:00.296337+0000","last_clean":"2024-03-16T11:04:00.296337+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:04:00.296337+0000","last_undegraded":"2024-03-16T11:04:00.296337+0000","last_fullsized":"2024-03-16T11:04:00.296337+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:17.910862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.291493+0000","last_change":"2024-03-16T11:04:00.290455+0000","last_active":"2024-03-16T11:04:00.291493+0000","last_peered":"2024-03-16T11:04:00.291493+0000","last_clean":"2024-03-16T11:04:00.291493+0000","last_became_active":"2024-03-16T11:04:00.290057+0000","last_became_peered":"2024-03-16T11:04:00.290057+0000","last_unstale":"2024-03-16T11:04:00.291493+0000","last_undegraded":"2024-03-16T11:04:00.291493+0000","last_fullsized":"2024-03-16T11:04:00.291493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:33.368456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.304226+0000","last_change":"2024-03-16T11:03:59.285059+0000","last_active":"2024-03-16T11:04:00.304226+0000","last_peered":"2024-03-16T11:04:00.304226+0000","last_clean":"2024-03-16T11:04:00.304226+0000","last_became_active":"2024-03-16T11:03:59.284745+0000","last_became_peered":"2024-03-16T11:03:59.284745+0000","last_unstale":"2024-03-16T11:04:00.304226+0000","last_undegraded":"2024-03-16T11:04:00.304226+0000","last_fullsized":"2024-03-16T11:04:00.304226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:12.516958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292685+0000","last_change":"2024-03-16T11:04:00.292933+0000","last_active":"2024-03-16T11:04:01.292685+0000","last_peered":"2024-03-16T11:04:01.292685+0000","last_clean":"2024-03-16T11:04:01.292685+0000","last_became_active":"2024-03-16T11:04:00.292637+0000","last_became_peered":"2024-03-16T11:04:00.292637+0000","last_unstale":"2024-03-16T11:04:01.292685+0000","last_undegraded":"2024-03-16T11:04:01.292685+0000","last_fullsized":"2024-03-16T11:04:01.292685+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:22.231735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292837+0000","last_change":"2024-03-16T11:03:59.285267+0000","last_active":"2024-03-16T11:04:01.292837+0000","last_peered":"2024-03-16T11:04:01.292837+0000","last_clean":"2024-03-16T11:04:01.292837+0000","last_became_active":"2024-03-16T11:03:59.284890+0000","last_became_peered":"2024-03-16T11:03:59.284890+0000","last_unstale":"2024-03-16T11:04:01.292837+0000","last_undegraded":"2024-03-16T11:04:01.292837+0000","last_fullsized":"2024-03-16T11:04:01.292837+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:53.625612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211339+0000","last_change":"2024-03-16T11:04:00.292660+0000","last_active":"2024-03-16T11:04:02.211339+0000","last_peered":"2024-03-16T11:04:02.211339+0000","last_clean":"2024-03-16T11:04:02.211339+0000","last_became_active":"2024-03-16T11:04:00.292366+0000","last_became_peered":"2024-03-16T11:04:00.292366+0000","last_unstale":"2024-03-16T11:04:02.211339+0000","last_undegraded":"2024-03-16T11:04:02.211339+0000","last_fullsized":"2024-03-16T11:04:02.211339+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:36:44.331456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293550+0000","last_change":"2024-03-16T11:03:59.286370+0000","last_active":"2024-03-16T11:04:01.293550+0000","last_peered":"2024-03-16T11:04:01.293550+0000","last_clean":"2024-03-16T11:04:01.293550+0000","last_became_active":"2024-03-16T11:03:59.286065+0000","last_became_peered":"2024-03-16T11:03:59.286065+0000","last_unstale":"2024-03-16T11:04:01.293550+0000","last_undegraded":"2024-03-16T11:04:01.293550+0000","last_fullsized":"2024-03-16T11:04:01.293550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:20.126848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210412+0000","last_change":"2024-03-16T11:04:00.290864+0000","last_active":"2024-03-16T11:04:02.210412+0000","last_peered":"2024-03-16T11:04:02.210412+0000","last_clean":"2024-03-16T11:04:02.210412+0000","last_became_active":"2024-03-16T11:04:00.290537+0000","last_became_peered":"2024-03-16T11:04:00.290537+0000","last_unstale":"2024-03-16T11:04:02.210412+0000","last_undegraded":"2024-03-16T11:04:02.210412+0000","last_fullsized":"2024-03-16T11:04:02.210412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:52.991767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293423+0000","last_change":"2024-03-16T11:03:59.281663+0000","last_active":"2024-03-16T11:04:01.293423+0000","last_peered":"2024-03-16T11:04:01.293423+0000","last_clean":"2024-03-16T11:04:01.293423+0000","last_became_active":"2024-03-16T11:03:59.281328+0000","last_became_peered":"2024-03-16T11:03:59.281328+0000","last_unstale":"2024-03-16T11:04:01.293423+0000","last_undegraded":"2024-03-16T11:04:01.293423+0000","last_fullsized":"2024-03-16T11:04:01.293423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:44.511109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010964+0000","last_change":"2024-03-16T11:04:00.297607+0000","last_active":"2024-03-16T11:04:02.010964+0000","last_peered":"2024-03-16T11:04:02.010964+0000","last_clean":"2024-03-16T11:04:02.010964+0000","last_became_active":"2024-03-16T11:04:00.297014+0000","last_became_peered":"2024-03-16T11:04:00.297014+0000","last_unstale":"2024-03-16T11:04:02.010964+0000","last_undegraded":"2024-03-16T11:04:02.010964+0000","last_fullsized":"2024-03-16T11:04:02.010964+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:07:33.726437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.472340+0000","last_change":"2024-03-16T11:03:59.286300+0000","last_active":"2024-03-16T11:04:03.472340+0000","last_peered":"2024-03-16T11:04:03.472340+0000","last_clean":"2024-03-16T11:04:03.472340+0000","last_became_active":"2024-03-16T11:03:59.285764+0000","last_became_peered":"2024-03-16T11:03:59.285764+0000","last_unstale":"2024-03-16T11:04:03.472340+0000","last_undegraded":"2024-03-16T11:04:03.472340+0000","last_fullsized":"2024-03-16T11:04:03.472340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:49.084245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293106+0000","last_change":"2024-03-16T11:04:00.289152+0000","last_active":"2024-03-16T11:04:01.293106+0000","last_peered":"2024-03-16T11:04:01.293106+0000","last_clean":"2024-03-16T11:04:01.293106+0000","last_became_active":"2024-03-16T11:04:00.288837+0000","last_became_peered":"2024-03-16T11:04:00.288837+0000","last_unstale":"2024-03-16T11:04:01.293106+0000","last_undegraded":"2024-03-16T11:04:01.293106+0000","last_fullsized":"2024-03-16T11:04:01.293106+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:45:01.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.312605+0000","last_change":"2024-03-16T11:03:59.281505+0000","last_active":"2024-03-16T11:04:03.312605+0000","last_peered":"2024-03-16T11:04:03.312605+0000","last_clean":"2024-03-16T11:04:03.312605+0000","last_became_active":"2024-03-16T11:03:59.281175+0000","last_became_peered":"2024-03-16T11:03:59.281175+0000","last_unstale":"2024-03-16T11:04:03.312605+0000","last_undegraded":"2024-03-16T11:04:03.312605+0000","last_fullsized":"2024-03-16T11:04:03.312605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:12.463914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010775+0000","last_change":"2024-03-16T11:04:00.297769+0000","last_active":"2024-03-16T11:04:02.010775+0000","last_peered":"2024-03-16T11:04:02.010775+0000","last_clean":"2024-03-16T11:04:02.010775+0000","last_became_active":"2024-03-16T11:04:00.297061+0000","last_became_peered":"2024-03-16T11:04:00.297061+0000","last_unstale":"2024-03-16T11:04:02.010775+0000","last_undegraded":"2024-03-16T11:04:02.010775+0000","last_fullsized":"2024-03-16T11:04:02.010775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:42.594315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.314859+0000","last_change":"2024-03-16T11:03:59.290260+0000","last_active":"2024-03-16T11:04:03.314859+0000","last_peered":"2024-03-16T11:04:03.314859+0000","last_clean":"2024-03-16T11:04:03.314859+0000","last_became_active":"2024-03-16T11:03:59.289843+0000","last_became_peered":"2024-03-16T11:03:59.289843+0000","last_unstale":"2024-03-16T11:04:03.314859+0000","last_undegraded":"2024-03-16T11:04:03.314859+0000","last_fullsized":"2024-03-16T11:04:03.314859+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:22.483488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292645+0000","last_change":"2024-03-16T11:04:00.287873+0000","last_active":"2024-03-16T11:04:01.292645+0000","last_peered":"2024-03-16T11:04:01.292645+0000","last_clean":"2024-03-16T11:04:01.292645+0000","last_became_active":"2024-03-16T11:04:00.287002+0000","last_became_peered":"2024-03-16T11:04:00.287002+0000","last_unstale":"2024-03-16T11:04:01.292645+0000","last_undegraded":"2024-03-16T11:04:01.292645+0000","last_fullsized":"2024-03-16T11:04:01.292645+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:57.851726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010567+0000","last_change":"2024-03-16T11:03:59.284745+0000","last_active":"2024-03-16T11:04:02.010567+0000","last_peered":"2024-03-16T11:04:02.010567+0000","last_clean":"2024-03-16T11:04:02.010567+0000","last_became_active":"2024-03-16T11:03:59.284325+0000","last_became_peered":"2024-03-16T11:03:59.284325+0000","last_unstale":"2024-03-16T11:04:02.010567+0000","last_undegraded":"2024-03-16T11:04:02.010567+0000","last_fullsized":"2024-03-16T11:04:02.010567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.537787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.291120+0000","last_change":"2024-03-16T11:04:00.290698+0000","last_active":"2024-03-16T11:04:00.291120+0000","last_peered":"2024-03-16T11:04:00.291120+0000","last_clean":"2024-03-16T11:04:00.291120+0000","last_became_active":"2024-03-16T11:04:00.290348+0000","last_became_peered":"2024-03-16T11:04:00.290348+0000","last_unstale":"2024-03-16T11:04:00.291120+0000","last_undegraded":"2024-03-16T11:04:00.291120+0000","last_fullsized":"2024-03-16T11:04:00.291120+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:49:02.626376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.470716+0000","last_change":"2024-03-16T11:03:59.280705+0000","last_active":"2024-03-16T11:04:03.470716+0000","last_peered":"2024-03-16T11:04:03.470716+0000","last_clean":"2024-03-16T11:04:03.470716+0000","last_became_active":"2024-03-16T11:03:59.280372+0000","last_became_peered":"2024-03-16T11:03:59.280372+0000","last_unstale":"2024-03-16T11:04:03.470716+0000","last_undegraded":"2024-03-16T11:04:03.470716+0000","last_fullsized":"2024-03-16T11:04:03.470716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:54.187322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210790+0000","last_change":"2024-03-16T11:04:00.293867+0000","last_active":"2024-03-16T11:04:02.210790+0000","last_peered":"2024-03-16T11:04:02.210790+0000","last_clean":"2024-03-16T11:04:02.210790+0000","last_became_active":"2024-03-16T11:04:00.293330+0000","last_became_peered":"2024-03-16T11:04:00.293330+0000","last_unstale":"2024-03-16T11:04:02.210790+0000","last_undegraded":"2024-03-16T11:04:02.210790+0000","last_fullsized":"2024-03-16T11:04:02.210790+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:54.798346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310441+0000","last_change":"2024-03-16T11:03:59.284202+0000","last_active":"2024-03-16T11:04:02.310441+0000","last_peered":"2024-03-16T11:04:02.310441+0000","last_clean":"2024-03-16T11:04:02.310441+0000","last_became_active":"2024-03-16T11:03:59.283656+0000","last_became_peered":"2024-03-16T11:03:59.283656+0000","last_unstale":"2024-03-16T11:04:02.310441+0000","last_undegraded":"2024-03-16T11:04:02.310441+0000","last_fullsized":"2024-03-16T11:04:02.310441+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:44.760307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211278+0000","last_change":"2024-03-16T11:04:00.301753+0000","last_active":"2024-03-16T11:04:02.211278+0000","last_peered":"2024-03-16T11:04:02.211278+0000","last_clean":"2024-03-16T11:04:02.211278+0000","last_became_active":"2024-03-16T11:04:00.301425+0000","last_became_peered":"2024-03-16T11:04:00.301425+0000","last_unstale":"2024-03-16T11:04:02.211278+0000","last_undegraded":"2024-03-16T11:04:02.211278+0000","last_fullsized":"2024-03-16T11:04:02.211278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:30.889952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309722+0000","last_change":"2024-03-16T11:03:59.288268+0000","last_active":"2024-03-16T11:04:02.309722+0000","last_peered":"2024-03-16T11:04:02.309722+0000","last_clean":"2024-03-16T11:04:02.309722+0000","last_became_active":"2024-03-16T11:03:59.287861+0000","last_became_peered":"2024-03-16T11:03:59.287861+0000","last_unstale":"2024-03-16T11:04:02.309722+0000","last_undegraded":"2024-03-16T11:04:02.309722+0000","last_fullsized":"2024-03-16T11:04:02.309722+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:34:34.248408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210678+0000","last_change":"2024-03-16T11:04:00.306490+0000","last_active":"2024-03-16T11:04:02.210678+0000","last_peered":"2024-03-16T11:04:02.210678+0000","last_clean":"2024-03-16T11:04:02.210678+0000","last_became_active":"2024-03-16T11:04:00.305874+0000","last_became_peered":"2024-03-16T11:04:00.305874+0000","last_unstale":"2024-03-16T11:04:02.210678+0000","last_undegraded":"2024-03-16T11:04:02.210678+0000","last_fullsized":"2024-03-16T11:04:02.210678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'11","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:04.830366+0000","last_change":"2024-03-16T11:03:59.280714+0000","last_active":"2024-03-16T11:04:04.830366+0000","last_peered":"2024-03-16T11:04:04.830366+0000","last_clean":"2024-03-16T11:04:04.830366+0000","last_became_active":"2024-03-16T11:03:59.280398+0000","last_became_peered":"2024-03-16T11:03:59.280398+0000","last_unstale":"2024-03-16T11:04:04.830366+0000","last_undegraded":"2024-03-16T11:04:04.830366+0000","last_fullsized":"2024-03-16T11:04:04.830366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:34.256125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293439+0000","last_change":"2024-03-16T11:04:00.285669+0000","last_active":"2024-03-16T11:04:01.293439+0000","last_peered":"2024-03-16T11:04:01.293439+0000","last_clean":"2024-03-16T11:04:01.293439+0000","last_became_active":"2024-03-16T11:04:00.285337+0000","last_became_peered":"2024-03-16T11:04:00.285337+0000","last_unstale":"2024-03-16T11:04:01.293439+0000","last_undegraded":"2024-03-16T11:04:01.293439+0000","last_fullsized":"2024-03-16T11:04:01.293439+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:36.925350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.306876+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:04:00.306876+0000","last_peered":"2024-03-16T11:04:00.306876+0000","last_clean":"2024-03-16T11:04:00.306876+0000","last_became_active":"2024-03-16T11:03:59.289357+0000","last_became_peered":"2024-03-16T11:03:59.289357+0000","last_unstale":"2024-03-16T11:04:00.306876+0000","last_undegraded":"2024-03-16T11:04:00.306876+0000","last_fullsized":"2024-03-16T11:04:00.306876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:26:35.860091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210766+0000","last_change":"2024-03-16T11:04:00.298488+0000","last_active":"2024-03-16T11:04:02.210766+0000","last_peered":"2024-03-16T11:04:02.210766+0000","last_clean":"2024-03-16T11:04:02.210766+0000","last_became_active":"2024-03-16T11:04:00.298202+0000","last_became_peered":"2024-03-16T11:04:00.298202+0000","last_unstale":"2024-03-16T11:04:02.210766+0000","last_undegraded":"2024-03-16T11:04:02.210766+0000","last_fullsized":"2024-03-16T11:04:02.210766+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:48.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310110+0000","last_change":"2024-03-16T11:03:59.287066+0000","last_active":"2024-03-16T11:04:02.310110+0000","last_peered":"2024-03-16T11:04:02.310110+0000","last_clean":"2024-03-16T11:04:02.310110+0000","last_became_active":"2024-03-16T11:03:59.286731+0000","last_became_peered":"2024-03-16T11:03:59.286731+0000","last_unstale":"2024-03-16T11:04:02.310110+0000","last_undegraded":"2024-03-16T11:04:02.310110+0000","last_fullsized":"2024-03-16T11:04:02.310110+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:51.443833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211723+0000","last_change":"2024-03-16T11:04:00.294450+0000","last_active":"2024-03-16T11:04:02.211723+0000","last_peered":"2024-03-16T11:04:02.211723+0000","last_clean":"2024-03-16T11:04:02.211723+0000","last_became_active":"2024-03-16T11:04:00.293769+0000","last_became_peered":"2024-03-16T11:04:00.293769+0000","last_unstale":"2024-03-16T11:04:02.211723+0000","last_undegraded":"2024-03-16T11:04:02.211723+0000","last_fullsized":"2024-03-16T11:04:02.211723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:02.755907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.472065+0000","last_change":"2024-03-16T11:03:59.286197+0000","last_active":"2024-03-16T11:04:03.472065+0000","last_peered":"2024-03-16T11:04:03.472065+0000","last_clean":"2024-03-16T11:04:03.472065+0000","last_became_active":"2024-03-16T11:03:59.285757+0000","last_became_peered":"2024-03-16T11:03:59.285757+0000","last_unstale":"2024-03-16T11:04:03.472065+0000","last_undegraded":"2024-03-16T11:04:03.472065+0000","last_fullsized":"2024-03-16T11:04:03.472065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:47.868106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211359+0000","last_change":"2024-03-16T11:04:00.296709+0000","last_active":"2024-03-16T11:04:02.211359+0000","last_peered":"2024-03-16T11:04:02.211359+0000","last_clean":"2024-03-16T11:04:02.211359+0000","last_became_active":"2024-03-16T11:04:00.296372+0000","last_became_peered":"2024-03-16T11:04:00.296372+0000","last_unstale":"2024-03-16T11:04:02.211359+0000","last_undegraded":"2024-03-16T11:04:02.211359+0000","last_fullsized":"2024-03-16T11:04:02.211359+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:37.948866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'10","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310921+0000","last_change":"2024-03-16T11:03:59.283831+0000","last_active":"2024-03-16T11:04:02.310921+0000","last_peered":"2024-03-16T11:04:02.310921+0000","last_clean":"2024-03-16T11:04:02.310921+0000","last_became_active":"2024-03-16T11:03:59.283404+0000","last_became_peered":"2024-03-16T11:03:59.283404+0000","last_unstale":"2024-03-16T11:04:02.310921+0000","last_undegraded":"2024-03-16T11:04:02.310921+0000","last_fullsized":"2024-03-16T11:04:02.310921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:26:34.012597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010572+0000","last_change":"2024-03-16T11:04:00.299703+0000","last_active":"2024-03-16T11:04:02.010572+0000","last_peered":"2024-03-16T11:04:02.010572+0000","last_clean":"2024-03-16T11:04:02.010572+0000","last_became_active":"2024-03-16T11:04:00.299321+0000","last_became_peered":"2024-03-16T11:04:00.299321+0000","last_unstale":"2024-03-16T11:04:02.010572+0000","last_undegraded":"2024-03-16T11:04:02.010572+0000","last_fullsized":"2024-03-16T11:04:02.010572+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.529281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293373+0000","last_change":"2024-03-16T11:03:59.277837+0000","last_active":"2024-03-16T11:04:01.293373+0000","last_peered":"2024-03-16T11:04:01.293373+0000","last_clean":"2024-03-16T11:04:01.293373+0000","last_became_active":"2024-03-16T11:03:59.277515+0000","last_became_peered":"2024-03-16T11:03:59.277515+0000","last_unstale":"2024-03-16T11:04:01.293373+0000","last_undegraded":"2024-03-16T11:04:01.293373+0000","last_fullsized":"2024-03-16T11:04:01.293373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:05.116990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292720+0000","last_change":"2024-03-16T11:04:00.292998+0000","last_active":"2024-03-16T11:04:01.292720+0000","last_peered":"2024-03-16T11:04:01.292720+0000","last_clean":"2024-03-16T11:04:01.292720+0000","last_became_active":"2024-03-16T11:04:00.292608+0000","last_became_peered":"2024-03-16T11:04:00.292608+0000","last_unstale":"2024-03-16T11:04:01.292720+0000","last_undegraded":"2024-03-16T11:04:01.292720+0000","last_fullsized":"2024-03-16T11:04:01.292720+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:41.996826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.276099+0000","last_change":"2024-03-16T11:03:59.283896+0000","last_active":"2024-03-16T11:04:00.276099+0000","last_peered":"2024-03-16T11:04:00.276099+0000","last_clean":"2024-03-16T11:04:00.276099+0000","last_became_active":"2024-03-16T11:03:59.283562+0000","last_became_peered":"2024-03-16T11:03:59.283562+0000","last_unstale":"2024-03-16T11:04:00.276099+0000","last_undegraded":"2024-03-16T11:04:00.276099+0000","last_fullsized":"2024-03-16T11:04:00.276099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:15:02.466629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288131+0000","last_change":"2024-03-16T11:04:00.288897+0000","last_active":"2024-03-16T11:04:01.288131+0000","last_peered":"2024-03-16T11:04:01.288131+0000","last_clean":"2024-03-16T11:04:01.288131+0000","last_became_active":"2024-03-16T11:04:00.288493+0000","last_became_peered":"2024-03-16T11:04:00.288493+0000","last_unstale":"2024-03-16T11:04:01.288131+0000","last_undegraded":"2024-03-16T11:04:01.288131+0000","last_fullsized":"2024-03-16T11:04:01.288131+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:31.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211778+0000","last_change":"2024-03-16T11:03:59.291595+0000","last_active":"2024-03-16T11:04:02.211778+0000","last_peered":"2024-03-16T11:04:02.211778+0000","last_clean":"2024-03-16T11:04:02.211778+0000","last_became_active":"2024-03-16T11:03:59.291159+0000","last_became_peered":"2024-03-16T11:03:59.291159+0000","last_unstale":"2024-03-16T11:04:02.211778+0000","last_undegraded":"2024-03-16T11:04:02.211778+0000","last_fullsized":"2024-03-16T11:04:02.211778+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:03.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.288103+0000","last_change":"2024-03-16T11:04:00.302356+0000","last_active":"2024-03-16T11:04:01.288103+0000","last_peered":"2024-03-16T11:04:01.288103+0000","last_clean":"2024-03-16T11:04:01.288103+0000","last_became_active":"2024-03-16T11:04:00.302025+0000","last_became_peered":"2024-03-16T11:04:00.302025+0000","last_unstale":"2024-03-16T11:04:01.288103+0000","last_undegraded":"2024-03-16T11:04:01.288103+0000","last_fullsized":"2024-03-16T11:04:01.288103+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:46.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010722+0000","last_change":"2024-03-16T11:04:00.298400+0000","last_active":"2024-03-16T11:04:02.010722+0000","last_peered":"2024-03-16T11:04:02.010722+0000","last_clean":"2024-03-16T11:04:02.010722+0000","last_became_active":"2024-03-16T11:04:00.298026+0000","last_became_peered":"2024-03-16T11:04:00.298026+0000","last_unstale":"2024-03-16T11:04:02.010722+0000","last_undegraded":"2024-03-16T11:04:02.010722+0000","last_fullsized":"2024-03-16T11:04:02.010722+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:08.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.313933+0000","last_change":"2024-03-16T11:03:59.287528+0000","last_active":"2024-03-16T11:04:03.313933+0000","last_peered":"2024-03-16T11:04:03.313933+0000","last_clean":"2024-03-16T11:04:03.313933+0000","last_became_active":"2024-03-16T11:03:59.287218+0000","last_became_peered":"2024-03-16T11:03:59.287218+0000","last_unstale":"2024-03-16T11:04:03.313933+0000","last_undegraded":"2024-03-16T11:04:03.313933+0000","last_fullsized":"2024-03-16T11:04:03.313933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:19.803233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.293020+0000","last_change":"2024-03-16T11:04:00.292857+0000","last_active":"2024-03-16T11:04:01.293020+0000","last_peered":"2024-03-16T11:04:01.293020+0000","last_clean":"2024-03-16T11:04:01.293020+0000","last_became_active":"2024-03-16T11:04:00.292574+0000","last_became_peered":"2024-03-16T11:04:00.292574+0000","last_unstale":"2024-03-16T11:04:01.293020+0000","last_undegraded":"2024-03-16T11:04:01.293020+0000","last_fullsized":"2024-03-16T11:04:01.293020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:48.466780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:03.469229+0000","last_change":"2024-03-16T11:03:59.281251+0000","last_active":"2024-03-16T11:04:03.469229+0000","last_peered":"2024-03-16T11:04:03.469229+0000","last_clean":"2024-03-16T11:04:03.469229+0000","last_became_active":"2024-03-16T11:03:59.280830+0000","last_became_peered":"2024-03-16T11:03:59.280830+0000","last_unstale":"2024-03-16T11:04:03.469229+0000","last_undegraded":"2024-03-16T11:04:03.469229+0000","last_fullsized":"2024-03-16T11:04:03.469229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:03:04.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":342,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211284+0000","last_change":"2024-03-16T11:02:48.497749+0000","last_active":"2024-03-16T11:04:02.211284+0000","last_peered":"2024-03-16T11:04:02.211284+0000","last_clean":"2024-03-16T11:04:02.211284+0000","last_became_active":"2024-03-16T11:02:48.497157+0000","last_became_peered":"2024-03-16T11:02:48.497157+0000","last_unstale":"2024-03-16T11:04:02.211284+0000","last_undegraded":"2024-03-16T11:04:02.211284+0000","last_fullsized":"2024-03-16T11:04:02.211284+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_clean_scrub_stamp":"2024-03-16T10:59:26.309639+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:04.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210680+0000","last_change":"2024-03-16T11:04:00.300821+0000","last_active":"2024-03-16T11:04:02.210680+0000","last_peered":"2024-03-16T11:04:02.210680+0000","last_clean":"2024-03-16T11:04:02.210680+0000","last_became_active":"2024-03-16T11:04:00.300510+0000","last_became_peered":"2024-03-16T11:04:00.300510+0000","last_unstale":"2024-03-16T11:04:02.210680+0000","last_undegraded":"2024-03-16T11:04:02.210680+0000","last_fullsized":"2024-03-16T11:04:02.210680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:57.035167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309011+0000","last_change":"2024-03-16T11:03:59.283609+0000","last_active":"2024-03-16T11:04:02.309011+0000","last_peered":"2024-03-16T11:04:02.309011+0000","last_clean":"2024-03-16T11:04:02.309011+0000","last_became_active":"2024-03-16T11:03:59.283227+0000","last_became_peered":"2024-03-16T11:03:59.283227+0000","last_unstale":"2024-03-16T11:04:02.309011+0000","last_undegraded":"2024-03-16T11:04:02.309011+0000","last_fullsized":"2024-03-16T11:04:02.309011+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:18.904202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309087+0000","last_change":"2024-03-16T11:03:59.283485+0000","last_active":"2024-03-16T11:04:02.309087+0000","last_peered":"2024-03-16T11:04:02.309087+0000","last_clean":"2024-03-16T11:04:02.309087+0000","last_became_active":"2024-03-16T11:03:59.283145+0000","last_became_peered":"2024-03-16T11:03:59.283145+0000","last_unstale":"2024-03-16T11:04:02.309087+0000","last_undegraded":"2024-03-16T11:04:02.309087+0000","last_fullsized":"2024-03-16T11:04:02.309087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:51:53.143512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210844+0000","last_change":"2024-03-16T11:04:00.285134+0000","last_active":"2024-03-16T11:04:02.210844+0000","last_peered":"2024-03-16T11:04:02.210844+0000","last_clean":"2024-03-16T11:04:02.210844+0000","last_became_active":"2024-03-16T11:04:00.284705+0000","last_became_peered":"2024-03-16T11:04:00.284705+0000","last_unstale":"2024-03-16T11:04:02.210844+0000","last_undegraded":"2024-03-16T11:04:02.210844+0000","last_fullsized":"2024-03-16T11:04:02.210844+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:38.637970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.276221+0000","last_change":"2024-03-16T11:03:59.289999+0000","last_active":"2024-03-16T11:04:00.276221+0000","last_peered":"2024-03-16T11:04:00.276221+0000","last_clean":"2024-03-16T11:04:00.276221+0000","last_became_active":"2024-03-16T11:03:59.289694+0000","last_became_peered":"2024-03-16T11:03:59.289694+0000","last_unstale":"2024-03-16T11:04:00.276221+0000","last_undegraded":"2024-03-16T11:04:00.276221+0000","last_fullsized":"2024-03-16T11:04:00.276221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:16.037387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292886+0000","last_change":"2024-03-16T11:04:00.298148+0000","last_active":"2024-03-16T11:04:01.292886+0000","last_peered":"2024-03-16T11:04:01.292886+0000","last_clean":"2024-03-16T11:04:01.292886+0000","last_became_active":"2024-03-16T11:04:00.297822+0000","last_became_peered":"2024-03-16T11:04:00.297822+0000","last_unstale":"2024-03-16T11:04:01.292886+0000","last_undegraded":"2024-03-16T11:04:01.292886+0000","last_fullsized":"2024-03-16T11:04:01.292886+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:09:28.645317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.308275+0000","last_change":"2024-03-16T11:03:59.292207+0000","last_active":"2024-03-16T11:04:02.308275+0000","last_peered":"2024-03-16T11:04:02.308275+0000","last_clean":"2024-03-16T11:04:02.308275+0000","last_became_active":"2024-03-16T11:03:59.291859+0000","last_became_peered":"2024-03-16T11:03:59.291859+0000","last_unstale":"2024-03-16T11:04:02.308275+0000","last_undegraded":"2024-03-16T11:04:02.308275+0000","last_fullsized":"2024-03-16T11:04:02.308275+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:55.365720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211254+0000","last_change":"2024-03-16T11:04:00.291261+0000","last_active":"2024-03-16T11:04:02.211254+0000","last_peered":"2024-03-16T11:04:02.211254+0000","last_clean":"2024-03-16T11:04:02.211254+0000","last_became_active":"2024-03-16T11:04:00.290832+0000","last_became_peered":"2024-03-16T11:04:00.290832+0000","last_unstale":"2024-03-16T11:04:02.211254+0000","last_undegraded":"2024-03-16T11:04:02.211254+0000","last_fullsized":"2024-03-16T11:04:02.211254+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:10.418808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.310908+0000","last_change":"2024-03-16T11:03:59.288014+0000","last_active":"2024-03-16T11:04:02.310908+0000","last_peered":"2024-03-16T11:04:02.310908+0000","last_clean":"2024-03-16T11:04:02.310908+0000","last_became_active":"2024-03-16T11:03:59.287679+0000","last_became_peered":"2024-03-16T11:03:59.287679+0000","last_unstale":"2024-03-16T11:04:02.310908+0000","last_undegraded":"2024-03-16T11:04:02.310908+0000","last_fullsized":"2024-03-16T11:04:02.310908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:16:39.156224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292913+0000","last_change":"2024-03-16T11:04:00.288047+0000","last_active":"2024-03-16T11:04:01.292913+0000","last_peered":"2024-03-16T11:04:01.292913+0000","last_clean":"2024-03-16T11:04:01.292913+0000","last_became_active":"2024-03-16T11:04:00.287619+0000","last_became_peered":"2024-03-16T11:04:00.287619+0000","last_unstale":"2024-03-16T11:04:01.292913+0000","last_undegraded":"2024-03-16T11:04:01.292913+0000","last_fullsized":"2024-03-16T11:04:01.292913+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:43.608349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292418+0000","last_change":"2024-03-16T11:03:59.288439+0000","last_active":"2024-03-16T11:04:01.292418+0000","last_peered":"2024-03-16T11:04:01.292418+0000","last_clean":"2024-03-16T11:04:01.292418+0000","last_became_active":"2024-03-16T11:03:59.288122+0000","last_became_peered":"2024-03-16T11:03:59.288122+0000","last_unstale":"2024-03-16T11:04:01.292418+0000","last_undegraded":"2024-03-16T11:04:01.292418+0000","last_fullsized":"2024-03-16T11:04:01.292418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:50:25.499608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292300+0000","last_change":"2024-03-16T11:04:00.290522+0000","last_active":"2024-03-16T11:04:01.292300+0000","last_peered":"2024-03-16T11:04:01.292300+0000","last_clean":"2024-03-16T11:04:01.292300+0000","last_became_active":"2024-03-16T11:04:00.290175+0000","last_became_peered":"2024-03-16T11:04:00.290175+0000","last_unstale":"2024-03-16T11:04:01.292300+0000","last_undegraded":"2024-03-16T11:04:01.292300+0000","last_fullsized":"2024-03-16T11:04:01.292300+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:34.113278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.279781+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:04:00.279781+0000","last_peered":"2024-03-16T11:04:00.279781+0000","last_clean":"2024-03-16T11:04:00.279781+0000","last_became_active":"2024-03-16T11:03:59.289367+0000","last_became_peered":"2024-03-16T11:03:59.289367+0000","last_unstale":"2024-03-16T11:04:00.279781+0000","last_undegraded":"2024-03-16T11:04:00.279781+0000","last_fullsized":"2024-03-16T11:04:00.279781+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:42:48.627116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292760+0000","last_change":"2024-03-16T11:04:00.287685+0000","last_active":"2024-03-16T11:04:01.292760+0000","last_peered":"2024-03-16T11:04:01.292760+0000","last_clean":"2024-03-16T11:04:01.292760+0000","last_became_active":"2024-03-16T11:04:00.287367+0000","last_became_peered":"2024-03-16T11:04:00.287367+0000","last_unstale":"2024-03-16T11:04:01.292760+0000","last_undegraded":"2024-03-16T11:04:01.292760+0000","last_fullsized":"2024-03-16T11:04:01.292760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:25:52.741651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.309713+0000","last_change":"2024-03-16T11:03:59.286293+0000","last_active":"2024-03-16T11:04:02.309713+0000","last_peered":"2024-03-16T11:04:02.309713+0000","last_clean":"2024-03-16T11:04:02.309713+0000","last_became_active":"2024-03-16T11:03:59.285817+0000","last_became_peered":"2024-03-16T11:03:59.285817+0000","last_unstale":"2024-03-16T11:04:02.309713+0000","last_undegraded":"2024-03-16T11:04:02.309713+0000","last_fullsized":"2024-03-16T11:04:02.309713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:38:12.093762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211897+0000","last_change":"2024-03-16T11:04:00.299391+0000","last_active":"2024-03-16T11:04:02.211897+0000","last_peered":"2024-03-16T11:04:02.211897+0000","last_clean":"2024-03-16T11:04:02.211897+0000","last_became_active":"2024-03-16T11:04:00.298931+0000","last_became_peered":"2024-03-16T11:04:00.298931+0000","last_unstale":"2024-03-16T11:04:02.211897+0000","last_undegraded":"2024-03-16T11:04:02.211897+0000","last_fullsized":"2024-03-16T11:04:02.211897+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:12:11.900154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.280431+0000","last_change":"2024-03-16T11:03:59.292319+0000","last_active":"2024-03-16T11:04:00.280431+0000","last_peered":"2024-03-16T11:04:00.280431+0000","last_clean":"2024-03-16T11:04:00.280431+0000","last_became_active":"2024-03-16T11:03:59.291953+0000","last_became_peered":"2024-03-16T11:03:59.291953+0000","last_unstale":"2024-03-16T11:04:00.280431+0000","last_undegraded":"2024-03-16T11:04:00.280431+0000","last_fullsized":"2024-03-16T11:04:00.280431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:41:04.615726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211686+0000","last_change":"2024-03-16T11:04:00.296544+0000","last_active":"2024-03-16T11:04:02.211686+0000","last_peered":"2024-03-16T11:04:02.211686+0000","last_clean":"2024-03-16T11:04:02.211686+0000","last_became_active":"2024-03-16T11:04:00.296078+0000","last_became_peered":"2024-03-16T11:04:00.296078+0000","last_unstale":"2024-03-16T11:04:02.211686+0000","last_undegraded":"2024-03-16T11:04:02.211686+0000","last_fullsized":"2024-03-16T11:04:02.211686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:56:22.846734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010914+0000","last_change":"2024-03-16T11:03:59.287172+0000","last_active":"2024-03-16T11:04:02.010914+0000","last_peered":"2024-03-16T11:04:02.010914+0000","last_clean":"2024-03-16T11:04:02.010914+0000","last_became_active":"2024-03-16T11:03:59.286785+0000","last_became_peered":"2024-03-16T11:03:59.286785+0000","last_unstale":"2024-03-16T11:04:02.010914+0000","last_undegraded":"2024-03-16T11:04:02.010914+0000","last_fullsized":"2024-03-16T11:04:02.010914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:17:55.584618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211458+0000","last_change":"2024-03-16T11:04:00.296019+0000","last_active":"2024-03-16T11:04:02.211458+0000","last_peered":"2024-03-16T11:04:02.211458+0000","last_clean":"2024-03-16T11:04:02.211458+0000","last_became_active":"2024-03-16T11:04:00.295685+0000","last_became_peered":"2024-03-16T11:04:00.295685+0000","last_unstale":"2024-03-16T11:04:02.211458+0000","last_undegraded":"2024-03-16T11:04:02.211458+0000","last_fullsized":"2024-03-16T11:04:02.211458+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:43.406124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210585+0000","last_change":"2024-03-16T11:03:59.288822+0000","last_active":"2024-03-16T11:04:02.210585+0000","last_peered":"2024-03-16T11:04:02.210585+0000","last_clean":"2024-03-16T11:04:02.210585+0000","last_became_active":"2024-03-16T11:03:59.288516+0000","last_became_peered":"2024-03-16T11:03:59.288516+0000","last_unstale":"2024-03-16T11:04:02.210585+0000","last_undegraded":"2024-03-16T11:04:02.210585+0000","last_fullsized":"2024-03-16T11:04:02.210585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:32:38.862476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211853+0000","last_change":"2024-03-16T11:04:00.296117+0000","last_active":"2024-03-16T11:04:02.211853+0000","last_peered":"2024-03-16T11:04:02.211853+0000","last_clean":"2024-03-16T11:04:02.211853+0000","last_became_active":"2024-03-16T11:04:00.295664+0000","last_became_peered":"2024-03-16T11:04:00.295664+0000","last_unstale":"2024-03-16T11:04:02.211853+0000","last_undegraded":"2024-03-16T11:04:02.211853+0000","last_fullsized":"2024-03-16T11:04:02.211853+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.448616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292441+0000","last_change":"2024-03-16T11:04:00.295035+0000","last_active":"2024-03-16T11:04:01.292441+0000","last_peered":"2024-03-16T11:04:01.292441+0000","last_clean":"2024-03-16T11:04:01.292441+0000","last_became_active":"2024-03-16T11:04:00.294689+0000","last_became_peered":"2024-03-16T11:04:00.294689+0000","last_unstale":"2024-03-16T11:04:01.292441+0000","last_undegraded":"2024-03-16T11:04:01.292441+0000","last_fullsized":"2024-03-16T11:04:01.292441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:55.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.294047+0000","last_change":"2024-03-16T11:04:00.292831+0000","last_active":"2024-03-16T11:04:00.294047+0000","last_peered":"2024-03-16T11:04:00.294047+0000","last_clean":"2024-03-16T11:04:00.294047+0000","last_became_active":"2024-03-16T11:04:00.292387+0000","last_became_peered":"2024-03-16T11:04:00.292387+0000","last_unstale":"2024-03-16T11:04:00.294047+0000","last_undegraded":"2024-03-16T11:04:00.294047+0000","last_fullsized":"2024-03-16T11:04:00.294047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:13:38.248585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.293620+0000","last_change":"2024-03-16T11:04:00.291267+0000","last_active":"2024-03-16T11:04:00.293620+0000","last_peered":"2024-03-16T11:04:00.293620+0000","last_clean":"2024-03-16T11:04:00.293620+0000","last_became_active":"2024-03-16T11:04:00.290911+0000","last_became_peered":"2024-03-16T11:04:00.290911+0000","last_unstale":"2024-03-16T11:04:00.293620+0000","last_undegraded":"2024-03-16T11:04:00.293620+0000","last_fullsized":"2024-03-16T11:04:00.293620+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:41:11.909490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.211280+0000","last_change":"2024-03-16T11:04:00.300470+0000","last_active":"2024-03-16T11:04:02.211280+0000","last_peered":"2024-03-16T11:04:02.211280+0000","last_clean":"2024-03-16T11:04:02.211280+0000","last_became_active":"2024-03-16T11:04:00.300107+0000","last_became_peered":"2024-03-16T11:04:00.300107+0000","last_unstale":"2024-03-16T11:04:02.211280+0000","last_undegraded":"2024-03-16T11:04:02.211280+0000","last_fullsized":"2024-03-16T11:04:02.211280+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:18.718749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210714+0000","last_change":"2024-03-16T11:04:00.300857+0000","last_active":"2024-03-16T11:04:02.210714+0000","last_peered":"2024-03-16T11:04:02.210714+0000","last_clean":"2024-03-16T11:04:02.210714+0000","last_became_active":"2024-03-16T11:04:00.300551+0000","last_became_peered":"2024-03-16T11:04:00.300551+0000","last_unstale":"2024-03-16T11:04:02.210714+0000","last_undegraded":"2024-03-16T11:04:02.210714+0000","last_fullsized":"2024-03-16T11:04:02.210714+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:24.362561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292613+0000","last_change":"2024-03-16T11:04:00.294060+0000","last_active":"2024-03-16T11:04:01.292613+0000","last_peered":"2024-03-16T11:04:01.292613+0000","last_clean":"2024-03-16T11:04:01.292613+0000","last_became_active":"2024-03-16T11:04:00.293615+0000","last_became_peered":"2024-03-16T11:04:00.293615+0000","last_unstale":"2024-03-16T11:04:01.292613+0000","last_undegraded":"2024-03-16T11:04:01.292613+0000","last_fullsized":"2024-03-16T11:04:01.292613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:28:33.789436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:04:00.297122+0000","last_change":"2024-03-16T11:04:00.294943+0000","last_active":"2024-03-16T11:04:00.297122+0000","last_peered":"2024-03-16T11:04:00.297122+0000","last_clean":"2024-03-16T11:04:00.297122+0000","last_became_active":"2024-03-16T11:04:00.294637+0000","last_became_peered":"2024-03-16T11:04:00.294637+0000","last_unstale":"2024-03-16T11:04:00.297122+0000","last_undegraded":"2024-03-16T11:04:00.297122+0000","last_fullsized":"2024-03-16T11:04:00.297122+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:37.630443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292728+0000","last_change":"2024-03-16T11:04:00.300737+0000","last_active":"2024-03-16T11:04:01.292728+0000","last_peered":"2024-03-16T11:04:01.292728+0000","last_clean":"2024-03-16T11:04:01.292728+0000","last_became_active":"2024-03-16T11:04:00.300410+0000","last_became_peered":"2024-03-16T11:04:00.300410+0000","last_unstale":"2024-03-16T11:04:01.292728+0000","last_undegraded":"2024-03-16T11:04:01.292728+0000","last_fullsized":"2024-03-16T11:04:01.292728+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:18.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.010710+0000","last_change":"2024-03-16T11:04:00.292958+0000","last_active":"2024-03-16T11:04:02.010710+0000","last_peered":"2024-03-16T11:04:02.010710+0000","last_clean":"2024-03-16T11:04:02.010710+0000","last_became_active":"2024-03-16T11:04:00.292599+0000","last_became_peered":"2024-03-16T11:04:00.292599+0000","last_unstale":"2024-03-16T11:04:02.010710+0000","last_undegraded":"2024-03-16T11:04:02.010710+0000","last_fullsized":"2024-03-16T11:04:02.010710+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:46.370597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292521+0000","last_change":"2024-03-16T11:04:00.291381+0000","last_active":"2024-03-16T11:04:01.292521+0000","last_peered":"2024-03-16T11:04:01.292521+0000","last_clean":"2024-03-16T11:04:01.292521+0000","last_became_active":"2024-03-16T11:04:00.291046+0000","last_became_peered":"2024-03-16T11:04:00.291046+0000","last_unstale":"2024-03-16T11:04:01.292521+0000","last_undegraded":"2024-03-16T11:04:01.292521+0000","last_fullsized":"2024-03-16T11:04:01.292521+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:08.185733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210591+0000","last_change":"2024-03-16T11:04:00.301381+0000","last_active":"2024-03-16T11:04:02.210591+0000","last_peered":"2024-03-16T11:04:02.210591+0000","last_clean":"2024-03-16T11:04:02.210591+0000","last_became_active":"2024-03-16T11:04:00.300990+0000","last_became_peered":"2024-03-16T11:04:00.300990+0000","last_unstale":"2024-03-16T11:04:02.210591+0000","last_undegraded":"2024-03-16T11:04:02.210591+0000","last_fullsized":"2024-03-16T11:04:02.210591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:07:43.775066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:02.210938+0000","last_change":"2024-03-16T11:04:00.298220+0000","last_active":"2024-03-16T11:04:02.210938+0000","last_peered":"2024-03-16T11:04:02.210938+0000","last_clean":"2024-03-16T11:04:02.210938+0000","last_became_active":"2024-03-16T11:04:00.297776+0000","last_became_peered":"2024-03-16T11:04:00.297776+0000","last_unstale":"2024-03-16T11:04:02.210938+0000","last_undegraded":"2024-03-16T11:04:02.210938+0000","last_fullsized":"2024-03-16T11:04:02.210938+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:40.387309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T11:04:01.292369+0000","last_change":"2024-03-16T11:04:00.292988+0000","last_active":"2024-03-16T11:04:01.292369+0000","last_peered":"2024-03-16T11:04:01.292369+0000","last_clean":"2024-03-16T11:04:01.292369+0000","last_became_active":"2024-03-16T11:04:00.292655+0000","last_became_peered":"2024-03-16T11:04:00.292655+0000","last_unstale":"2024-03-16T11:04:01.292369+0000","last_undegraded":"2024-03-16T11:04:01.292369+0000","last_fullsized":"2024-03-16T11:04:01.292369+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:33:43.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":11038,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":20225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285581,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":453300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547219,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1585152,"data_stored":1046858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":937984,"data_stored":445312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005084,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":453904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266721,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1552384,"data_stored":1044872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430247,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":453390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":958464,"data_stored":454912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":454778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":446376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":954368,"data_stored":453404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888111,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":453944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1568768,"data_stored":1045144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":10181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:04:07.539 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T11:04:07.540 INFO:tasks.cephadm:Setting up client nodes... 2024-03-16T11:04:07.540 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T11:04:07.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:07 smithi153 ceph-mon[35521]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:07.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:07 smithi153 ceph-mon[35521]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-16T11:04:07.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:07 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3314010773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:04:07.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:07 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4107787558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:07.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:07 smithi153 ceph-mon[35521]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:04:07.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:07 smithi078 ceph-mon[32235]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:07.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:07 smithi078 ceph-mon[32235]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-16T11:04:07.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:07 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3314010773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:04:07.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:07 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4107787558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:07.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:07 smithi078 ceph-mon[32235]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:04:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:07.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:07 smithi098 ceph-mon[35848]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:04:07.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:07 smithi098 ceph-mon[35848]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-16T11:04:07.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:07 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3314010773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:04:07.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:07 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4107787558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:07.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:07 smithi098 ceph-mon[35848]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:04:08.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:08 smithi078 ceph-mon[32235]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:04:08.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:08 smithi078 ceph-mon[32235]: pgmap v291: 97 pgs: 97 active+clean; 588 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5.9 KiB/s wr, 19 op/s 2024-03-16T11:04:08.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:08 smithi153 ceph-mon[35521]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:04:08.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:08 smithi153 ceph-mon[35521]: pgmap v291: 97 pgs: 97 active+clean; 588 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5.9 KiB/s wr, 19 op/s 2024-03-16T11:04:08.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:08 smithi098 ceph-mon[35848]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:04:08.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:08 smithi098 ceph-mon[35848]: pgmap v291: 97 pgs: 97 active+clean; 588 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5.9 KiB/s wr, 19 op/s 2024-03-16T11:04:09.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 -- 172.21.15.78:0/3471124278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 msgr2=0x7ff694102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 --2- 172.21.15.78:0/3471124278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff694102880 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0099c0 tx=0x7ff67c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 -- 172.21.15.78:0/3471124278 shutdown_connections 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 --2- 172.21.15.78:0/3471124278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff694102dc0 0x7ff694109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 --2- 172.21.15.78:0/3471124278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101220 0x7ff694101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 --2- 172.21.15.78:0/3471124278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff694102880 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 -- 172.21.15.78:0/3471124278 >> 172.21.15.78:0/3471124278 conn(0x7ff6940fc7c0 msgr2=0x7ff6940fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 -- 172.21.15.78:0/3471124278 shutdown_connections 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 -- 172.21.15.78:0/3471124278 wait complete. 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 Processor -- start 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.342+0000 7ff6999f7700 1 -- start start 2024-03-16T11:04:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6999f7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101220 0x7ff694197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6999f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff6941977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6999f7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff694102dc0 0x7ff69419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6999f7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff69410cc90 con 0x7ff694102420 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6999f7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff69410cb10 con 0x7ff694101220 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6999f7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff69410ce10 con 0x7ff694102dc0 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff68bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff6941977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff692ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101220 0x7ff694197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff68bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff6941977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44476/0 (socket says 172.21.15.78:44476) 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff68bfff700 1 -- 172.21.15.78:0/2556313258 learned_addr learned my addr 172.21.15.78:0/2556313258 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6937fe700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff694102dc0 0x7ff69419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff692ffd700 1 -- 172.21.15.78:0/2556313258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff694102dc0 msgr2=0x7ff69419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff692ffd700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff694102dc0 0x7ff69419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff692ffd700 1 -- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 msgr2=0x7ff6941977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff692ffd700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff6941977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff692ffd700 1 -- 172.21.15.78:0/2556313258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69419eee0 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff68bfff700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff6941977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff692ffd700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101220 0x7ff694197280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff68400e9c0 tx=0x7ff68400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6989f5700 1 -- 172.21.15.78:0/2556313258 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684004220 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69419f1d0 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6989f5700 1 -- 172.21.15.78:0/2556313258 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6840043c0 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.343+0000 7ff6989f5700 1 -- 172.21.15.78:0/2556313258 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684003c40 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.344+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69419f760 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.344+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6940507e0 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.345+0000 7ff6989f5700 1 -- 172.21.15.78:0/2556313258 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff68400c7c0 con 0x7ff694101220 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.346+0000 7ff6989f5700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff6800a29d0 0x7ff6800a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.346+0000 7ff68bfff700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff6800a29d0 0x7ff6800a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.346+0000 7ff6989f5700 1 -- 172.21.15.78:0/2556313258 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff6840cee70 con 0x7ff694101220 2024-03-16T11:04:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.346+0000 7ff68bfff700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff6800a29d0 0x7ff6800a4e90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0036f0 tx=0x7ff67c0037c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.354+0000 7ff6989f5700 1 -- 172.21.15.78:0/2556313258 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff68409bef0 con 0x7ff694101220 2024-03-16T11:04:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.632+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7ff69402dba0 con 0x7ff694101220 2024-03-16T11:04:09.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.636+0000 7ff6989f5700 1 -- 172.21.15.78:0/2556313258 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7ff68409b7f0 con 0x7ff694101220 2024-03-16T11:04:09.637 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-03-16T11:04:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCpfPVlrx/IJRAAEsn36w8GJliFVxi+yh65Uw== 2024-03-16T11:04:09.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff6800a29d0 msgr2=0x7ff6800a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:09.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff6800a29d0 0x7ff6800a4e90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0036f0 tx=0x7ff67c0037c0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101220 msgr2=0x7ff694197280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:09.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101220 0x7ff694197280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff68400e9c0 tx=0x7ff68400ecd0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 shutdown_connections 2024-03-16T11:04:09.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff6800a29d0 0x7ff6800a4e90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff694102dc0 0x7ff69419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff694101220 0x7ff694197280 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 --2- 172.21.15.78:0/2556313258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694102420 0x7ff6941977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 >> 172.21.15.78:0/2556313258 conn(0x7ff6940fc7c0 msgr2=0x7ff6941057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 shutdown_connections 2024-03-16T11:04:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:09.638+0000 7ff6999f7700 1 -- 172.21.15.78:0/2556313258 wait complete. 2024-03-16T11:04:09.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:09 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2556313258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:04:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:09 smithi078 ceph-mon[32235]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:04:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:09 smithi078 ceph-mon[32235]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T11:04:09.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:09 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2556313258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:04:09.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:09 smithi098 ceph-mon[35848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:04:09.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:09 smithi098 ceph-mon[35848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T11:04:10.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:09 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2556313258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:04:10.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:09 smithi153 ceph-mon[35521]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:04:10.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:09 smithi153 ceph-mon[35521]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T11:04:10.262 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T11:04:10.262 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-16T11:04:10.262 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T11:04:10.306 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T11:04:10.306 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-16T11:04:10.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-16T11:04:10.588 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:10.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:10 smithi098 ceph-mon[35848]: pgmap v292: 97 pgs: 97 active+clean; 589 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5.1 KiB/s wr, 17 op/s 2024-03-16T11:04:11.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:10 smithi078 ceph-mon[32235]: pgmap v292: 97 pgs: 97 active+clean; 589 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5.1 KiB/s wr, 17 op/s 2024-03-16T11:04:11.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:10 smithi153 ceph-mon[35521]: pgmap v292: 97 pgs: 97 active+clean; 589 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5.1 KiB/s wr, 17 op/s 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 -- 172.21.15.78:0/2388896722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 msgr2=0x7fd15010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 --2- 172.21.15.78:0/2388896722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd15010f160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd14800b0d0 tx=0x7fd14800b3e0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 -- 172.21.15.78:0/2388896722 shutdown_connections 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 --2- 172.21.15.78:0/2388896722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd150108320 0x7fd150108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 --2- 172.21.15.78:0/2388896722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd15010f160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 --2- 172.21.15.78:0/2388896722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150069f80 0x7fd15006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 -- 172.21.15.78:0/2388896722 >> 172.21.15.78:0/2388896722 conn(0x7fd150078100 msgr2=0x7fd150078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 -- 172.21.15.78:0/2388896722 shutdown_connections 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.228+0000 7fd1560ee700 1 -- 172.21.15.78:0/2388896722 wait complete. 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 Processor -- start 2024-03-16T11:04:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 -- start start 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd150069f80 0x7fd1501012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd1501017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150108320 0x7fd150101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd150112420 con 0x7fd15006a920 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501122a0 con 0x7fd150108320 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd1560ee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501125a0 con 0x7fd150069f80 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd146ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd1501017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd14f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd150069f80 0x7fd1501012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd146ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd1501017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44486/0 (socket says 172.21.15.78:44486) 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd14f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd150069f80 0x7fd1501012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:38302/0 (socket says 172.21.15.78:38302) 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd146ffd700 1 -- 172.21.15.78:0/3683480868 learned_addr learned my addr 172.21.15.78:0/3683480868 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.229+0000 7fd14ffff700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150108320 0x7fd150101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd146ffd700 1 -- 172.21.15.78:0/3683480868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd150069f80 msgr2=0x7fd1501012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd146ffd700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd150069f80 0x7fd1501012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd146ffd700 1 -- 172.21.15.78:0/3683480868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150108320 msgr2=0x7fd150101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd146ffd700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150108320 0x7fd150101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd146ffd700 1 -- 172.21.15.78:0/3683480868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1501a6e30 con 0x7fd15006a920 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd146ffd700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd1501017e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd13800e9a0 tx=0x7fd13800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd14d7fa700 1 -- 172.21.15.78:0/3683480868 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13800ca10 con 0x7fd15006a920 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd14d7fa700 1 -- 172.21.15.78:0/3683480868 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd13800cbb0 con 0x7fd15006a920 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1501a7120 con 0x7fd15006a920 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd14d7fa700 1 -- 172.21.15.78:0/3683480868 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd138018680 con 0x7fd15006a920 2024-03-16T11:04:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.230+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1501a7660 con 0x7fd15006a920 2024-03-16T11:04:12.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.231+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd150068f90 con 0x7fd15006a920 2024-03-16T11:04:12.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.232+0000 7fd14d7fa700 1 -- 172.21.15.78:0/3683480868 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd138010430 con 0x7fd15006a920 2024-03-16T11:04:12.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.233+0000 7fd14d7fa700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd13c0a2920 0x7fd13c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:12.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.233+0000 7fd14f7fe700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd13c0a2920 0x7fd13c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:12.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.233+0000 7fd14d7fa700 1 -- 172.21.15.78:0/3683480868 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fd1380cebc0 con 0x7fd15006a920 2024-03-16T11:04:12.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.233+0000 7fd14f7fe700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd13c0a2920 0x7fd13c0a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd140007b60 tx=0x7fd1400073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:12.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.238+0000 7fd14d7fa700 1 -- 172.21.15.78:0/3683480868 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd13809bcb0 con 0x7fd15006a920 2024-03-16T11:04:12.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.490+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd15002dd30 con 0x7fd15006a920 2024-03-16T11:04:12.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.495+0000 7fd14d7fa700 1 -- 172.21.15.78:0/3683480868 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262660 (secure 0 0 0) 0x7fd13809b5b0 con 0x7fd15006a920 2024-03-16T11:04:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:12.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd13c0a2920 msgr2=0x7fd13c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:12.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd13c0a2920 0x7fd13c0a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd140007b60 tx=0x7fd1400073d0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 msgr2=0x7fd1501017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd1501017e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd13800e9a0 tx=0x7fd13800ecb0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 shutdown_connections 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd13c0a2920 0x7fd13c0a4de0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd150069f80 0x7fd1501012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd150108320 0x7fd150101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 --2- 172.21.15.78:0/3683480868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd15006a920 0x7fd1501017e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 >> 172.21.15.78:0/3683480868 conn(0x7fd150078100 msgr2=0x7fd1500ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 shutdown_connections 2024-03-16T11:04:12.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:12.498+0000 7fd1560ee700 1 -- 172.21.15.78:0/3683480868 wait complete. 2024-03-16T11:04:13.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:13 smithi078 ceph-mon[32235]: pgmap v293: 97 pgs: 97 active+clean; 624 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-03-16T11:04:13.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:13 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3683480868' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T11:04:13.209 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":4254906951},{"type":"v1","addr":"172.21.15.78:6801","nonce":4254906951}]},"active_addr":"172.21.15.78:6801/4254906951","active_change":"2024-03-16T10:57:01.884129+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.78:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2941755181}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3714764002}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2346392144}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3417580197}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":688778073}]}]} 2024-03-16T11:04:13.215 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-16T11:04:13.215 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-16T11:04:13.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T11:04:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:13 smithi153 ceph-mon[35521]: pgmap v293: 97 pgs: 97 active+clean; 624 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-03-16T11:04:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:13 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3683480868' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T11:04:13.452 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:13 smithi098 ceph-mon[35848]: pgmap v293: 97 pgs: 97 active+clean; 624 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-03-16T11:04:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:13 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3683480868' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T11:04:15.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 -- 172.21.15.78:0/2738085123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 msgr2=0x7ffb9810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 --2- 172.21.15.78:0/2738085123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb9810f1e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffb900099c0 tx=0x7ffb90009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 -- 172.21.15.78:0/2738085123 shutdown_connections 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 --2- 172.21.15.78:0/2738085123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb981025b0 0x7ffb98102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 --2- 172.21.15.78:0/2738085123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb9810f1e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 --2- 172.21.15.78:0/2738085123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98102f60 0x7ffb98106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 -- 172.21.15.78:0/2738085123 >> 172.21.15.78:0/2738085123 conn(0x7ffb980793f0 msgr2=0x7ffb98079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 -- 172.21.15.78:0/2738085123 shutdown_connections 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 -- 172.21.15.78:0/2738085123 wait complete. 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 Processor -- start 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.104+0000 7ffba0475700 1 -- start start 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb981025b0 0x7ffb9819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:15.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb98102f60 0x7ffb9819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb98198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981124d0 con 0x7ffb98107660 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb98112350 con 0x7ffb981025b0 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffb98112650 con 0x7ffb98102f60 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb98198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb98198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44506/0 (socket says 172.21.15.78:44506) 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9e211700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb981025b0 0x7ffb9819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 -- 172.21.15.78:0/12431703 learned_addr learned my addr 172.21.15.78:0/12431703 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9da10700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb98102f60 0x7ffb9819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 -- 172.21.15.78:0/12431703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb98102f60 msgr2=0x7ffb9819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb98102f60 0x7ffb9819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 -- 172.21.15.78:0/12431703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb981025b0 msgr2=0x7ffb9819e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb981025b0 0x7ffb9819e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 -- 172.21.15.78:0/12431703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb98199050 con 0x7ffb98107660 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9e211700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb981025b0 0x7ffb9819e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb9ea12700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb98198770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffb900059d0 tx=0x7ffb90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb8f7fe700 1 -- 172.21.15.78:0/12431703 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90026070 con 0x7ffb98107660 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb8f7fe700 1 -- 172.21.15.78:0/12431703 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb90003c70 con 0x7ffb98107660 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb981992e0 con 0x7ffb98107660 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffb8f7fe700 1 -- 172.21.15.78:0/12431703 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9000fba0 con 0x7ffb98107660 2024-03-16T11:04:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.105+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb981a5450 con 0x7ffb98107660 2024-03-16T11:04:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.106+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb9810a810 con 0x7ffb98107660 2024-03-16T11:04:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.107+0000 7ffb8f7fe700 1 -- 172.21.15.78:0/12431703 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ffb90029070 con 0x7ffb98107660 2024-03-16T11:04:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.108+0000 7ffb8f7fe700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffb840a2920 0x7ffb840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.108+0000 7ffb9e211700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffb840a2920 0x7ffb840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.108+0000 7ffb8f7fe700 1 -- 172.21.15.78:0/12431703 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ffb900cf690 con 0x7ffb98107660 2024-03-16T11:04:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.108+0000 7ffb9e211700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffb840a2920 0x7ffb840a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffb94003e30 tx=0x7ffb9400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:15.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.111+0000 7ffb8f7fe700 1 -- 172.21.15.78:0/12431703 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb90030080 con 0x7ffb98107660 2024-03-16T11:04:15.112 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:15 smithi078 ceph-mon[32235]: pgmap v294: 97 pgs: 97 active+clean; 624 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-03-16T11:04:15.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:15 smithi153 ceph-mon[35521]: pgmap v294: 97 pgs: 97 active+clean; 624 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-03-16T11:04:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.311+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb98199ca0 con 0x7ffb98107660 2024-03-16T11:04:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.313+0000 7ffb8f7fe700 1 -- 172.21.15.78:0/12431703 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22136 (secure 0 0 0) 0x7ffb90013070 con 0x7ffb98107660 2024-03-16T11:04:15.314 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:15.314 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":88,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:04:01.279417+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffb840a2920 msgr2=0x7ffb840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffb840a2920 0x7ffb840a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffb94003e30 tx=0x7ffb9400b040 comp rx=0 tx=0).stop 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 msgr2=0x7ffb98198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb98198770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffb900059d0 tx=0x7ffb90004270 comp rx=0 tx=0).stop 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 shutdown_connections 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffb840a2920 0x7ffb840a4de0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb98107660 0x7ffb98198770 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb981025b0 0x7ffb9819e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 --2- 172.21.15.78:0/12431703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb98102f60 0x7ffb9819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 >> 172.21.15.78:0/12431703 conn(0x7ffb980793f0 msgr2=0x7ffb98101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 shutdown_connections 2024-03-16T11:04:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:15.315+0000 7ffba0475700 1 -- 172.21.15.78:0/12431703 wait complete. 2024-03-16T11:04:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:15 smithi098 ceph-mon[35848]: pgmap v294: 97 pgs: 97 active+clean; 624 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-03-16T11:04:15.892 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-16T11:04:15.892 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T11:04:16.134 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:16.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:16 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/12431703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:16.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:16 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/12431703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:16.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:16 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/12431703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:17.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:17 smithi078 ceph-mon[32235]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-03-16T11:04:17.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:17 smithi153 ceph-mon[35521]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-03-16T11:04:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:17 smithi098 ceph-mon[35848]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-03-16T11:04:17.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/425949249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806b420 msgr2=0x7ff9d8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:17.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/425949249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806b420 0x7ff9d8106f00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009990 tx=0x7ff9c0009ca0 comp rx=0 tx=0).stop 2024-03-16T11:04:17.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/425949249 shutdown_connections 2024-03-16T11:04:17.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/425949249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d806aa70 0x7ff9d806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:17.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/425949249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d81075e0 0x7ff9d810f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:17.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/425949249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806b420 0x7ff9d8106f00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:17.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/425949249 >> 172.21.15.78:0/425949249 conn(0x7ff9d8077220 msgr2=0x7ff9d8077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:17.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/425949249 shutdown_connections 2024-03-16T11:04:17.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.808+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/425949249 wait complete. 2024-03-16T11:04:17.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 Processor -- start 2024-03-16T11:04:17.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 -- start start 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806aa70 0x7ff9d819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d806b420 0x7ff9d819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d81075e0 0x7ff9d8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d8112430 con 0x7ff9d806aa70 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d81122b0 con 0x7ff9d806b420 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9ddfd0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d81125b0 con 0x7ff9d81075e0 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9d77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806aa70 0x7ff9d819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9d7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d81075e0 0x7ff9d8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9d77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806aa70 0x7ff9d819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59672/0 (socket says 172.21.15.78:59672) 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9d77fe700 1 -- 172.21.15.78:0/4285830843 learned_addr learned my addr 172.21.15.78:0/4285830843 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.809+0000 7ff9d6ffd700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d806b420 0x7ff9d819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d7fff700 1 -- 172.21.15.78:0/4285830843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d806b420 msgr2=0x7ff9d819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d7fff700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d806b420 0x7ff9d819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:17.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d7fff700 1 -- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806aa70 msgr2=0x7ff9d819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d7fff700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806aa70 0x7ff9d819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d7fff700 1 -- 172.21.15.78:0/4285830843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d8198ff0 con 0x7ff9d81075e0 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d77fe700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806aa70 0x7ff9d819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d7fff700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d81075e0 0x7ff9d8198810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800e970 tx=0x7ff9c800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d4ff9700 1 -- 172.21.15.78:0/4285830843 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8004220 con 0x7ff9d81075e0 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d4ff9700 1 -- 172.21.15.78:0/4285830843 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9c80043c0 con 0x7ff9d81075e0 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d81992e0 con 0x7ff9d81075e0 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9d4ff9700 1 -- 172.21.15.78:0/4285830843 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8010700 con 0x7ff9d81075e0 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.810+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d81a54a0 con 0x7ff9d81075e0 2024-03-16T11:04:17.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.811+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d81039e0 con 0x7ff9d81075e0 2024-03-16T11:04:17.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.812+0000 7ff9d4ff9700 1 -- 172.21.15.78:0/4285830843 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff9c8004750 con 0x7ff9d81075e0 2024-03-16T11:04:17.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.813+0000 7ff9d4ff9700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9c40a2920 0x7ff9c40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:17.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.813+0000 7ff9d77fe700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9c40a2920 0x7ff9c40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:17.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.813+0000 7ff9d4ff9700 1 -- 172.21.15.78:0/4285830843 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff9c80ce7c0 con 0x7ff9d81075e0 2024-03-16T11:04:17.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.813+0000 7ff9d77fe700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9c40a2920 0x7ff9c40a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009c10 tx=0x7ff9d000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:17.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:17.816+0000 7ff9d4ff9700 1 -- 172.21.15.78:0/4285830843 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9c809b8b0 con 0x7ff9d81075e0 2024-03-16T11:04:18.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.016+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9d802db70 con 0x7ff9d81075e0 2024-03-16T11:04:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.017+0000 7ff9d4ff9700 1 -- 172.21.15.78:0/4285830843 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22136 (secure 0 0 0) 0x7ff9c809b1b0 con 0x7ff9d81075e0 2024-03-16T11:04:18.018 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:04:18.018 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":88,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:04:01.279417+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9c40a2920 msgr2=0x7ff9c40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9c40a2920 0x7ff9c40a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009c10 tx=0x7ff9d000b040 comp rx=0 tx=0).stop 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d81075e0 msgr2=0x7ff9d8198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d81075e0 0x7ff9d8198810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800e970 tx=0x7ff9c800ec80 comp rx=0 tx=0).stop 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 shutdown_connections 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9c40a2920 0x7ff9c40a4de0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d81075e0 0x7ff9d8198810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d806aa70 0x7ff9d819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 --2- 172.21.15.78:0/4285830843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d806b420 0x7ff9d819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 >> 172.21.15.78:0/4285830843 conn(0x7ff9d8077220 msgr2=0x7ff9d80ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 shutdown_connections 2024-03-16T11:04:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:18.020+0000 7ff9ddfd0700 1 -- 172.21.15.78:0/4285830843 wait complete. 2024-03-16T11:04:18.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:18 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4285830843' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:18.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:18 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4285830843' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:18 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4285830843' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:04:18.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-16T11:04:18.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-16T11:04:18.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-16T11:04:18.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-16T11:04:18.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-16T11:04:18.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-16T11:04:18.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-16T11:04:18.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-16T11:04:18.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-16T11:04:18.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-16T11:04:18.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-16T11:04:18.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-16T11:04:19.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:19 smithi078 ceph-mon[32235]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-16T11:04:19.281 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:19.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:19 smithi153 ceph-mon[35521]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-16T11:04:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:19 smithi098 ceph-mon[35848]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-16T11:04:20.366 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.369 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.375 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.376 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.376 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.376 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.376 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.378 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.378 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.380 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:20.390 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:21.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:21 smithi078 ceph-mon[32235]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-03-16T11:04:21.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:21 smithi153 ceph-mon[35521]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 -- 172.21.15.78:0/248109034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc390102510 msgr2=0x7fc3901028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 --2- 172.21.15.78:0/248109034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc390102510 0x7fc3901028f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc3780099c0 tx=0x7fc378009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 -- 172.21.15.78:0/248109034 shutdown_connections 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 --2- 172.21.15.78:0/248109034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3901075f0 0x7fc39010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 --2- 172.21.15.78:0/248109034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc390102510 0x7fc3901028f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 --2- 172.21.15.78:0/248109034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc390102ec0 0x7fc390106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 -- 172.21.15.78:0/248109034 >> 172.21.15.78:0/248109034 conn(0x7fc3900fdd80 msgr2=0x7fc3901001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 -- 172.21.15.78:0/248109034 shutdown_connections 2024-03-16T11:04:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.312+0000 7fc396312700 1 -- 172.21.15.78:0/248109034 wait complete. 2024-03-16T11:04:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 Processor -- start 2024-03-16T11:04:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 -- start start 2024-03-16T11:04:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc390102510 0x7fc39019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc390102ec0 0x7fc39019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3901075f0 0x7fc390198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc390112480 con 0x7fc3901075f0 2024-03-16T11:04:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc390112300 con 0x7fc390102ec0 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc396312700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc390112600 con 0x7fc390102510 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc3948af700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3901075f0 0x7fc390198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc390102510 0x7fc39019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc390102510 0x7fc39019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:40214/0 (socket says 172.21.15.78:40214) 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc3948af700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3901075f0 0x7fc390198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59690/0 (socket says 172.21.15.78:59690) 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38ffff700 1 -- 172.21.15.78:0/3907267325 learned_addr learned my addr 172.21.15.78:0/3907267325 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc390102ec0 0x7fc39019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38ffff700 1 -- 172.21.15.78:0/3907267325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc390102ec0 msgr2=0x7fc39019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38ffff700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc390102ec0 0x7fc39019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38ffff700 1 -- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3901075f0 msgr2=0x7fc390198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.313+0000 7fc38ffff700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3901075f0 0x7fc390198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc38ffff700 1 -- 172.21.15.78:0/3907267325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc390199040 con 0x7fc390102510 2024-03-16T11:04:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc3948af700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3901075f0 0x7fc390198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc38ffff700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc390102510 0x7fc39019e690 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc378006010 tx=0x7fc378004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc378026070 con 0x7fc390102510 2024-03-16T11:04:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc378004630 con 0x7fc390102510 2024-03-16T11:04:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3901992d0 con 0x7fc390102510 2024-03-16T11:04:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37800fb20 con 0x7fc390102510 2024-03-16T11:04:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.314+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3901a54f0 con 0x7fc390102510 2024-03-16T11:04:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.315+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3900629e0 con 0x7fc390102510 2024-03-16T11:04:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.316+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc378029070 con 0x7fc390102510 2024-03-16T11:04:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.316+0000 7fc38d7fa700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc37c0a29a0 0x7fc37c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc38f7fe700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc37c0a29a0 0x7fc37c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fc3780cf900 con 0x7fc390102510 2024-03-16T11:04:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc38d7fa700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] conn(0x7fc37c0ac050 0x7fc37c0ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 --> [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc37801ae50 con 0x7fc37c0ac050 2024-03-16T11:04:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fc3780cfcb0 con 0x7fc390102510 2024-03-16T11:04:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc38f7fe700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc37c0a29a0 0x7fc37c0a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc3900fe1a0 tx=0x7fc380009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc3948af700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] conn(0x7fc37c0ac050 0x7fc37c0ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.317+0000 7fc3948af700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] conn(0x7fc37c0ac050 0x7fc37c0ae470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.319+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== osd.10 v2:172.21.15.98:6824/652996874 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc37801ae50 con 0x7fc37c0ac050 2024-03-16T11:04:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.334+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 --> [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc390062de0 con 0x7fc37c0ac050 2024-03-16T11:04:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc38d7fa700 1 -- 172.21.15.78:0/3907267325 <== osd.10 v2:172.21.15.98:6824/652996874 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc390062de0 con 0x7fc37c0ac050 2024-03-16T11:04:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 >> [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] conn(0x7fc37c0ac050 msgr2=0x7fc37c0ae470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] conn(0x7fc37c0ac050 0x7fc37c0ae470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc37c0a29a0 msgr2=0x7fc37c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc37c0a29a0 0x7fc37c0a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc3900fe1a0 tx=0x7fc380009260 comp rx=0 tx=0).stop 2024-03-16T11:04:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc390102510 msgr2=0x7fc39019e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 --2- 172.21.15.78:0/3907267325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc390102510 0x7fc39019e690 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc378006010 tx=0x7fc378004270 comp rx=0 tx=0).stop 2024-03-16T11:04:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc3948af700 1 -- 172.21.15.78:0/3907267325 reap_dead start 2024-03-16T11:04:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 shutdown_connections 2024-03-16T11:04:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 >> 172.21.15.78:0/3907267325 conn(0x7fc3900fdd80 msgr2=0x7fc3900ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 shutdown_connections 2024-03-16T11:04:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:21.335+0000 7fc396312700 1 -- 172.21.15.78:0/3907267325 wait complete. 2024-03-16T11:04:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:21 smithi098 ceph-mon[35848]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-03-16T11:04:23.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:23 smithi078 ceph-mon[32235]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-16T11:04:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:23 smithi098 ceph-mon[35848]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-16T11:04:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:23 smithi153 ceph-mon[35521]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-16T11:04:25.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:25 smithi098 ceph-mon[35848]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T11:04:25.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:25 smithi078 ceph-mon[32235]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T11:04:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:25 smithi153 ceph-mon[35521]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T11:04:27.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:27 smithi078 ceph-mon[32235]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-16T11:04:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:27 smithi098 ceph-mon[35848]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-16T11:04:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:27 smithi153 ceph-mon[35521]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-16T11:04:29.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:29 smithi078 ceph-mon[32235]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:04:29.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:29 smithi098 ceph-mon[35848]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:04:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:29 smithi153 ceph-mon[35521]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:04:31.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:31 smithi078 ceph-mon[32235]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:04:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:31 smithi098 ceph-mon[35848]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:04:31.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 -- 172.21.15.78:0/2318077274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79ac069f80 msgr2=0x7f79ac06a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:31.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 --2- 172.21.15.78:0/2318077274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79ac069f80 0x7f79ac06a3e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79a00099c0 tx=0x7f79a0009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:31.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 -- 172.21.15.78:0/2318077274 shutdown_connections 2024-03-16T11:04:31.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 --2- 172.21.15.78:0/2318077274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79ac06a920 0x7f79ac10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:31.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 --2- 172.21.15.78:0/2318077274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac108320 0x7f79ac108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:31.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 --2- 172.21.15.78:0/2318077274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79ac069f80 0x7f79ac06a3e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:31.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 -- 172.21.15.78:0/2318077274 >> 172.21.15.78:0/2318077274 conn(0x7f79ac078100 msgr2=0x7f79ac078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:31.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 -- 172.21.15.78:0/2318077274 shutdown_connections 2024-03-16T11:04:31.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.531+0000 7f79b100d700 1 -- 172.21.15.78:0/2318077274 wait complete. 2024-03-16T11:04:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.532+0000 7f79b100d700 1 Processor -- start 2024-03-16T11:04:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.532+0000 7f79b100d700 1 -- start start 2024-03-16T11:04:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79b100d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79ac069f80 0x7f79ac19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79b100d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06a920 0x7f79ac19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79b100d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79ac108320 0x7f79ac1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79b100d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac112450 con 0x7f79ac108320 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79b100d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1122d0 con 0x7f79ac06a920 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79b100d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1125d0 con 0x7f79ac069f80 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06a920 0x7f79ac19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06a920 0x7f79ac19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:42840/0 (socket says 172.21.15.78:42840) 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 -- 172.21.15.78:0/2109996485 learned_addr learned my addr 172.21.15.78:0/2109996485 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79ac069f80 0x7f79ac19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79ab59e700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79ac108320 0x7f79ac1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 -- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79ac069f80 msgr2=0x7f79ac19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79ac069f80 0x7f79ac19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 -- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79ac108320 msgr2=0x7f79ac1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79ac108320 0x7f79ac1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 -- 172.21.15.78:0/2109996485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac199090 con 0x7f79ac06a920 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aad9d700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79ac069f80 0x7f79ac19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.533+0000 7f79aa59c700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06a920 0x7f79ac19ebd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f79a0006010 tx=0x7f79a0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.534+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0026070 con 0x7f79ac06a920 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.534+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79a0004630 con 0x7f79ac06a920 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.534+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a000fb20 con 0x7f79ac06a920 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.534+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ac199320 con 0x7f79ac06a920 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.534+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ac1a5440 con 0x7f79ac06a920 2024-03-16T11:04:31.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.535+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7998001030 con 0x7f79ac06a920 2024-03-16T11:04:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.536+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f79a0029070 con 0x7f79ac06a920 2024-03-16T11:04:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.537+0000 7f7993fff700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f79940a29a0 0x7f79940a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.537+0000 7f79aad9d700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f79940a29a0 0x7f79940a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.537+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f79a00cf9f0 con 0x7f79ac06a920 2024-03-16T11:04:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.538+0000 7f7993fff700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] conn(0x7f79940ac180 0x7f79940ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.538+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 --> [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79a0015ec0 con 0x7f79940ac180 2024-03-16T11:04:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.538+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f79a00cfe50 con 0x7f79ac06a920 2024-03-16T11:04:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.538+0000 7f79ab59e700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] conn(0x7f79940ac180 0x7f79940ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.538+0000 7f79aad9d700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f79940a29a0 0x7f79940a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f799c009fd0 tx=0x7f799c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.538+0000 7f79ab59e700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] conn(0x7f79940ac180 0x7f79940ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.539+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== osd.8 v2:172.21.15.153:6816/208983464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f79a0015ec0 con 0x7f79940ac180 2024-03-16T11:04:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:31 smithi153 ceph-mon[35521]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:04:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.555+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 --> [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7998002e60 con 0x7f79940ac180 2024-03-16T11:04:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.555+0000 7f7993fff700 1 -- 172.21.15.78:0/2109996485 <== osd.8 v2:172.21.15.153:6816/208983464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7998002e60 con 0x7f79940ac180 2024-03-16T11:04:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] conn(0x7f79940ac180 msgr2=0x7f79940ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] conn(0x7f79940ac180 0x7f79940ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f79940a29a0 msgr2=0x7f79940a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f79940a29a0 0x7f79940a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f799c009fd0 tx=0x7f799c009260 comp rx=0 tx=0).stop 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06a920 msgr2=0x7f79ac19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 --2- 172.21.15.78:0/2109996485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06a920 0x7f79ac19ebd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f79a0006010 tx=0x7f79a0004270 comp rx=0 tx=0).stop 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79ab59e700 1 -- 172.21.15.78:0/2109996485 reap_dead start 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 shutdown_connections 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 >> 172.21.15.78:0/2109996485 conn(0x7f79ac078100 msgr2=0x7f79ac0ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 shutdown_connections 2024-03-16T11:04:31.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:31.556+0000 7f79b100d700 1 -- 172.21.15.78:0/2109996485 wait complete. 2024-03-16T11:04:32.962 INFO:teuthology.orchestra.run.smithi078.stdout:261993005090 2024-03-16T11:04:32.963 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-16T11:04:33.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:33 smithi078 ceph-mon[32235]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:04:33.317 INFO:teuthology.orchestra.run.smithi078.stdout:322122547223 2024-03-16T11:04:33.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-16T11:04:33.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:33 smithi098 ceph-mon[35848]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:04:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:33 smithi153 ceph-mon[35521]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:04:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 -- 172.21.15.78:0/3118587740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 msgr2=0x7f50841075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 --2- 172.21.15.78:0/3118587740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 0x7f50841075f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5080009990 tx=0x7f5080009ca0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 -- 172.21.15.78:0/3118587740 shutdown_connections 2024-03-16T11:04:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 --2- 172.21.15.78:0/3118587740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5084101b70 0x7f508410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 --2- 172.21.15.78:0/3118587740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 0x7f50841075f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 --2- 172.21.15.78:0/3118587740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50841011d0 0x7f5084101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 -- 172.21.15.78:0/3118587740 >> 172.21.15.78:0/3118587740 conn(0x7f5084078100 msgr2=0x7f5084078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 -- 172.21.15.78:0/3118587740 shutdown_connections 2024-03-16T11:04:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.948+0000 7f508bfa6700 1 -- 172.21.15.78:0/3118587740 wait complete. 2024-03-16T11:04:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 Processor -- start 2024-03-16T11:04:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 -- start start 2024-03-16T11:04:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50841011d0 0x7f50841975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5084101b70 0x7f5084197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 0x7f5084198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5084111300 con 0x7f5084107210 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5084111180 con 0x7f50841011d0 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508bfa6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5084111480 con 0x7f5084101b70 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 0x7f5084198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 0x7f5084198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41032/0 (socket says 172.21.15.78:41032) 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 -- 172.21.15.78:0/1689947783 learned_addr learned my addr 172.21.15.78:0/1689947783 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f5089541700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5084101b70 0x7f5084197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f5089d42700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50841011d0 0x7f50841975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 -- 172.21.15.78:0/1689947783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5084101b70 msgr2=0x7f5084197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5084101b70 0x7f5084197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 -- 172.21.15.78:0/1689947783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50841011d0 msgr2=0x7f50841975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50841011d0 0x7f50841975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 -- 172.21.15.78:0/1689947783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50841a6e20 con 0x7f5084107210 2024-03-16T11:04:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f5089d42700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50841011d0 0x7f50841975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.949+0000 7f508a543700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 0x7f5084198030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f507c007ee0 tx=0x7f507c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.950+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c00dd20 con 0x7f5084107210 2024-03-16T11:04:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.950+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f507c004020 con 0x7f5084107210 2024-03-16T11:04:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.950+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50841a7110 con 0x7f5084107210 2024-03-16T11:04:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.950+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c015660 con 0x7f5084107210 2024-03-16T11:04:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.950+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50841a7600 con 0x7f5084107210 2024-03-16T11:04:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.951+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5084062a60 con 0x7f5084107210 2024-03-16T11:04:33.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.952+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f507c0041d0 con 0x7f5084107210 2024-03-16T11:04:33.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.952+0000 7f507affd700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f50700a2920 0x7f50700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:33.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.952+0000 7f5089d42700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f50700a2920 0x7f50700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:33.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.953+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f507c0ceea0 con 0x7f5084107210 2024-03-16T11:04:33.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.953+0000 7f507affd700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] conn(0x7f50700ac100 0x7f50700ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:33.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.953+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 --> [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50700aebd0 con 0x7f50700ac100 2024-03-16T11:04:33.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.953+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f507c0cf380 con 0x7f5084107210 2024-03-16T11:04:33.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.953+0000 7f5089541700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] conn(0x7f50700ac100 0x7f50700ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:33.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.953+0000 7f5089d42700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f50700a2920 0x7f50700a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f50800098e0 tx=0x7f5080005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:33.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.953+0000 7f5089541700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] conn(0x7f50700ac100 0x7f50700ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:33.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.954+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== osd.3 v2:172.21.15.78:6810/3489770309 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50700aebd0 con 0x7f50700ac100 2024-03-16T11:04:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 --> [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5084062ee0 con 0x7f50700ac100 2024-03-16T11:04:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f507affd700 1 -- 172.21.15.78:0/1689947783 <== osd.3 v2:172.21.15.78:6810/3489770309 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5084062ee0 con 0x7f50700ac100 2024-03-16T11:04:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] conn(0x7f50700ac100 msgr2=0x7f50700ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] conn(0x7f50700ac100 0x7f50700ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f50700a2920 msgr2=0x7f50700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f50700a2920 0x7f50700a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f50800098e0 tx=0x7f5080005960 comp rx=0 tx=0).stop 2024-03-16T11:04:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 msgr2=0x7f5084198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 --2- 172.21.15.78:0/1689947783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5084107210 0x7f5084198030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f507c007ee0 tx=0x7f507c00d3b0 comp rx=0 tx=0).stop 2024-03-16T11:04:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508a543700 1 -- 172.21.15.78:0/1689947783 reap_dead start 2024-03-16T11:04:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 shutdown_connections 2024-03-16T11:04:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.970+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 >> 172.21.15.78:0/1689947783 conn(0x7f5084078100 msgr2=0x7f50841000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.971+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 shutdown_connections 2024-03-16T11:04:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:33.971+0000 7f508bfa6700 1 -- 172.21.15.78:0/1689947783 wait complete. 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 -- 172.21.15.78:0/1349465801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb006b420 msgr2=0x7efeb0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 --2- 172.21.15.78:0/1349465801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb006b420 0x7efeb0106f00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efea40098f0 tx=0x7efea4009c00 comp rx=0 tx=0).stop 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 -- 172.21.15.78:0/1349465801 shutdown_connections 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 --2- 172.21.15.78:0/1349465801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb01075e0 0x7efeb010f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 --2- 172.21.15.78:0/1349465801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeb006aa70 0x7efeb006ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 --2- 172.21.15.78:0/1349465801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb006b420 0x7efeb0106f00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 -- 172.21.15.78:0/1349465801 >> 172.21.15.78:0/1349465801 conn(0x7efeb0077220 msgr2=0x7efeb0077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 -- 172.21.15.78:0/1349465801 shutdown_connections 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 -- 172.21.15.78:0/1349465801 wait complete. 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.723+0000 7efeb5f04700 1 Processor -- start 2024-03-16T11:04:34.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeb5f04700 1 -- start start 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeb5f04700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb006aa70 0x7efeb01012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeb5f04700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeb006b420 0x7efeb0101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeb5f04700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb01075e0 0x7efeb0101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeb5f04700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efeb0112430 con 0x7efeb006b420 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeb5f04700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efeb01122b0 con 0x7efeb01075e0 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeb5f04700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efeb01125b0 con 0x7efeb006aa70 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeb006b420 0x7efeb0101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeb006b420 0x7efeb0101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41060/0 (socket says 172.21.15.78:41060) 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 -- 172.21.15.78:0/3864891543 learned_addr learned my addr 172.21.15.78:0/3864891543 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaf7fe700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb006aa70 0x7efeb01012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaffff700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb01075e0 0x7efeb0101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 -- 172.21.15.78:0/3864891543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb006aa70 msgr2=0x7efeb01012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb006aa70 0x7efeb01012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:34.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 -- 172.21.15.78:0/3864891543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb01075e0 msgr2=0x7efeb0101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:34.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb01075e0 0x7efeb0101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:34.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaeffd700 1 -- 172.21.15.78:0/3864891543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb01a6ed0 con 0x7efeb006b420 2024-03-16T11:04:34.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaf7fe700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb006aa70 0x7efeb01012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:34.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.724+0000 7efeaffff700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeb01075e0 0x7efeb0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:34.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.725+0000 7efeaeffd700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeb006b420 0x7efeb0101810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efea4009230 tx=0x7efea400f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:34.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.725+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea40050c0 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.725+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efea400b560 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.725+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb01a7160 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.725+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4003a40 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.725+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb01a75b0 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.726+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efeb0062a60 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.727+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efea400ba80 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeacff9700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe980a2920 0x7efe980a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeaf7fe700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe980a2920 0x7efe980a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7efea40cff10 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeacff9700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] conn(0x7efe980ac100 0x7efe980ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 --> [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efea400b0b0 con 0x7efe980ac100 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7efea40d03f0 con 0x7efeb006b420 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeaffff700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] conn(0x7efe980ac100 0x7efe980ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeaf7fe700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe980a2920 0x7efe980a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efea0009750 tx=0x7efea0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:34.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.728+0000 7efeaffff700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] conn(0x7efe980ac100 0x7efe980ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:34.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.729+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== osd.0 v2:172.21.15.78:6802/467332989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efea400b0b0 con 0x7efe980ac100 2024-03-16T11:04:34.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.745+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 --> [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efeb0062ee0 con 0x7efe980ac100 2024-03-16T11:04:34.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.745+0000 7efeacff9700 1 -- 172.21.15.78:0/3864891543 <== osd.0 v2:172.21.15.78:6802/467332989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efeb0062ee0 con 0x7efe980ac100 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] conn(0x7efe980ac100 msgr2=0x7efe980ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] conn(0x7efe980ac100 0x7efe980ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe980a2920 msgr2=0x7efe980a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe980a2920 0x7efe980a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efea0009750 tx=0x7efea0009310 comp rx=0 tx=0).stop 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeb006b420 msgr2=0x7efeb0101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 --2- 172.21.15.78:0/3864891543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeb006b420 0x7efeb0101810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efea4009230 tx=0x7efea400f990 comp rx=0 tx=0).stop 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeaffff700 1 -- 172.21.15.78:0/3864891543 reap_dead start 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 shutdown_connections 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 >> 172.21.15.78:0/3864891543 conn(0x7efeb0077220 msgr2=0x7efeb01087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 shutdown_connections 2024-03-16T11:04:34.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.746+0000 7efeb5f04700 1 -- 172.21.15.78:0/3864891543 wait complete. 2024-03-16T11:04:35.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 -- 172.21.15.78:0/1249096090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 msgr2=0x7f36101028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 --2- 172.21.15.78:0/1249096090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 0x7f36101028f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f360c0099c0 tx=0x7f360c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 -- 172.21.15.78:0/1249096090 shutdown_connections 2024-03-16T11:04:35.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 --2- 172.21.15.78:0/1249096090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101075f0 0x7f361010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 --2- 172.21.15.78:0/1249096090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 0x7f36101028f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 --2- 172.21.15.78:0/1249096090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3610102ec0 0x7f3610106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 -- 172.21.15.78:0/1249096090 >> 172.21.15.78:0/1249096090 conn(0x7f36100fdd80 msgr2=0x7f36101001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:35.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 -- 172.21.15.78:0/1249096090 shutdown_connections 2024-03-16T11:04:35.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.182+0000 7f36185a3700 1 -- 172.21.15.78:0/1249096090 wait complete. 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 Processor -- start 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 -- start start 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 0x7f361019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3610102ec0 0x7f361019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101075f0 0x7f36101987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3610112480 con 0x7f3610102510 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3610112300 con 0x7f36101075f0 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f36185a3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3610112600 con 0x7f3610102ec0 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f361633f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 0x7f361019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f361633f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 0x7f361019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41070/0 (socket says 172.21.15.78:41070) 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f361633f700 1 -- 172.21.15.78:0/2202879241 learned_addr learned my addr 172.21.15.78:0/2202879241 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:35.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f3616b40700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101075f0 0x7f36101987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.183+0000 7f3615b3e700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3610102ec0 0x7f361019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f361633f700 1 -- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3610102ec0 msgr2=0x7f361019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f361633f700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3610102ec0 0x7f361019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f361633f700 1 -- 172.21.15.78:0/2202879241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101075f0 msgr2=0x7f36101987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f361633f700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101075f0 0x7f36101987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f361633f700 1 -- 172.21.15.78:0/2202879241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3610198fa0 con 0x7f3610102510 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f3616b40700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101075f0 0x7f36101987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f361633f700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 0x7f361019e640 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f360c006010 tx=0x7f360c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c026070 con 0x7f3610102510 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3610199230 con 0x7f3610102510 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36101a54a0 con 0x7f3610102510 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f360c004630 con 0x7f3610102510 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.184+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c00fb20 con 0x7f3610102510 2024-03-16T11:04:35.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.185+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35f4001030 con 0x7f3610102510 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.187+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f360c029070 con 0x7f3610102510 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.188+0000 7f36077fe700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35fc0a29a0 0x7f35fc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.188+0000 7f3615b3e700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35fc0a29a0 0x7f35fc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.188+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f360c0cfa50 con 0x7f3610102510 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.189+0000 7f36077fe700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] conn(0x7f35fc0ac180 0x7f35fc0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.189+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 --> [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f360c00b0b0 con 0x7f35fc0ac180 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.189+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f360c0cff30 con 0x7f3610102510 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.189+0000 7f3616b40700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] conn(0x7f35fc0ac180 0x7f35fc0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.189+0000 7f3615b3e700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35fc0a29a0 0x7f35fc0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f36100ff1a0 tx=0x7f3600009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.189+0000 7f3616b40700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] conn(0x7f35fc0ac180 0x7f35fc0ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.190+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== osd.2 v2:172.21.15.153:6800/2715466096 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f360c00b0b0 con 0x7f35fc0ac180 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.207+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 --> [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f35f4002e60 con 0x7f35fc0ac180 2024-03-16T11:04:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.207+0000 7f36077fe700 1 -- 172.21.15.78:0/2202879241 <== osd.2 v2:172.21.15.153:6800/2715466096 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f35f4002e60 con 0x7f35fc0ac180 2024-03-16T11:04:35.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] conn(0x7f35fc0ac180 msgr2=0x7f35fc0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] conn(0x7f35fc0ac180 0x7f35fc0ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35fc0a29a0 msgr2=0x7f35fc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35fc0a29a0 0x7f35fc0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f36100ff1a0 tx=0x7f3600009260 comp rx=0 tx=0).stop 2024-03-16T11:04:35.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 msgr2=0x7f361019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 --2- 172.21.15.78:0/2202879241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3610102510 0x7f361019e640 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f360c006010 tx=0x7f360c004270 comp rx=0 tx=0).stop 2024-03-16T11:04:35.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f3616b40700 1 -- 172.21.15.78:0/2202879241 reap_dead start 2024-03-16T11:04:35.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 shutdown_connections 2024-03-16T11:04:35.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 >> 172.21.15.78:0/2202879241 conn(0x7f36100fdd80 msgr2=0x7f3610106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:35.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 shutdown_connections 2024-03-16T11:04:35.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:35.208+0000 7f36185a3700 1 -- 172.21.15.78:0/2202879241 wait complete. 2024-03-16T11:04:35.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:35 smithi078 ceph-mon[32235]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:04:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:35 smithi098 ceph-mon[35848]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:04:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:35 smithi153 ceph-mon[35521]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:04:35.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 -- 172.21.15.78:0/2309133424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48108360 msgr2=0x7fef48108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 --2- 172.21.15.78:0/2309133424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48108360 0x7fef48108740 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fef380099c0 tx=0x7fef38009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 -- 172.21.15.78:0/2309133424 shutdown_connections 2024-03-16T11:04:35.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 --2- 172.21.15.78:0/2309133424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef48102d10 0x7fef4810f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 --2- 172.21.15.78:0/2309133424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48108360 0x7fef48108740 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 --2- 172.21.15.78:0/2309133424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef48102350 0x7fef481027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 -- 172.21.15.78:0/2309133424 >> 172.21.15.78:0/2309133424 conn(0x7fef48079430 msgr2=0x7fef48079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 -- 172.21.15.78:0/2309133424 shutdown_connections 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.619+0000 7fef4ea5b700 1 -- 172.21.15.78:0/2309133424 wait complete. 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 Processor -- start 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 -- start start 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48102350 0x7fef480745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef48102d10 0x7fef48074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef48108360 0x7fef48075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fef481124c0 con 0x7fef48102350 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef48112340 con 0x7fef48102d10 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.620+0000 7fef4ea5b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef48112640 con 0x7fef48108360 2024-03-16T11:04:35.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48102350 0x7fef480745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48102350 0x7fef480745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41038/0 (socket says 172.21.15.78:41038) 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 -- 172.21.15.78:0/1121590985 learned_addr learned my addr 172.21.15.78:0/1121590985 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef4cff8700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef48108360 0x7fef48075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 -- 172.21.15.78:0/1121590985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef48108360 msgr2=0x7fef48075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef48108360 0x7fef48075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 -- 172.21.15.78:0/1121590985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef48102d10 msgr2=0x7fef48074b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef48102d10 0x7fef48074b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 -- 172.21.15.78:0/1121590985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef481a6ee0 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef47fff700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48102350 0x7fef480745d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef38006010 tx=0x7fef38004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38026070 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef38003c70 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.621+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3800fc20 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.622+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef481a7170 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.622+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef481a7570 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.623+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fef480629a0 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.624+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fef38029070 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.624+0000 7fef457fa700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fef300a29a0 0x7fef300a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef477fe700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fef300a29a0 0x7fef300a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fef380cfdc0 con 0x7fef48102350 2024-03-16T11:04:35.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef457fa700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] conn(0x7fef300ac180 0x7fef300ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 --> [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef300aec50 con 0x7fef300ac180 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fef380d0170 con 0x7fef48102350 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef477fe700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fef300a29a0 0x7fef300a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fef48079b00 tx=0x7fef3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef4cff8700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] conn(0x7fef300ac180 0x7fef300ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.625+0000 7fef4cff8700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] conn(0x7fef300ac180 0x7fef300ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.626+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== osd.1 v2:172.21.15.98:6800/64709971 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fef300aec50 con 0x7fef300ac180 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.642+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 --> [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fef48062dd0 con 0x7fef300ac180 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef457fa700 1 -- 172.21.15.78:0/1121590985 <== osd.1 v2:172.21.15.98:6800/64709971 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fef48062dd0 con 0x7fef300ac180 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 >> [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] conn(0x7fef300ac180 msgr2=0x7fef300ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] conn(0x7fef300ac180 0x7fef300ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fef300a29a0 msgr2=0x7fef300a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fef300a29a0 0x7fef300a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fef48079b00 tx=0x7fef3c009260 comp rx=0 tx=0).stop 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48102350 msgr2=0x7fef480745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:35.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 --2- 172.21.15.78:0/1121590985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef48102350 0x7fef480745d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef38006010 tx=0x7fef38004320 comp rx=0 tx=0).stop 2024-03-16T11:04:35.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4cff8700 1 -- 172.21.15.78:0/1121590985 reap_dead start 2024-03-16T11:04:35.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 shutdown_connections 2024-03-16T11:04:35.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 >> 172.21.15.78:0/1121590985 conn(0x7fef48079430 msgr2=0x7fef48101010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:35.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 shutdown_connections 2024-03-16T11:04:35.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:34.643+0000 7fef4ea5b700 1 -- 172.21.15.78:0/1121590985 wait complete. 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 -- 172.21.15.78:0/2843368546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc1011d0 msgr2=0x7fc7cc101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 --2- 172.21.15.78:0/2843368546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc1011d0 0x7fc7cc101630 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009990 tx=0x7fc7c0009ca0 comp rx=0 tx=0).stop 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 -- 172.21.15.78:0/2843368546 shutdown_connections 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 --2- 172.21.15.78:0/2843368546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7cc101b70 0x7fc7cc10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 --2- 172.21.15.78:0/2843368546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7cc107210 0x7fc7cc1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 --2- 172.21.15.78:0/2843368546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc1011d0 0x7fc7cc101630 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 -- 172.21.15.78:0/2843368546 >> 172.21.15.78:0/2843368546 conn(0x7fc7cc078100 msgr2=0x7fc7cc078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 -- 172.21.15.78:0/2843368546 shutdown_connections 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 -- 172.21.15.78:0/2843368546 wait complete. 2024-03-16T11:04:36.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.862+0000 7fc7d14da700 1 Processor -- start 2024-03-16T11:04:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7d14da700 1 -- start start 2024-03-16T11:04:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7d14da700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7cc1011d0 0x7fc7cc199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7d14da700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7cc101b70 0x7fc7cc199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7d14da700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc107210 0x7fc7cc1a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7d14da700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc111340 con 0x7fc7cc107210 2024-03-16T11:04:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7d14da700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc1111c0 con 0x7fc7cc1011d0 2024-03-16T11:04:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7d14da700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc1114c0 con 0x7fc7cc101b70 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc107210 0x7fc7cc1a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc107210 0x7fc7cc1a48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38712/0 (socket says 172.21.15.78:38712) 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 -- 172.21.15.78:0/1639603624 learned_addr learned my addr 172.21.15.78:0/1639603624 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7caffd700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7cc1011d0 0x7fc7cc199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7ca7fc700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7cc101b70 0x7fc7cc199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 -- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7cc101b70 msgr2=0x7fc7cc199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7cc101b70 0x7fc7cc199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 -- 172.21.15.78:0/1639603624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7cc1011d0 msgr2=0x7fc7cc199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7cc1011d0 0x7fc7cc199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 -- 172.21.15.78:0/1639603624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7cc1a4e30 con 0x7fc7cc107210 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7cb7fe700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc107210 0x7fc7cc1a48f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400e940 tx=0x7fc7c400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.863+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4004a30 con 0x7fc7cc107210 2024-03-16T11:04:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.864+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c40054d0 con 0x7fc7cc107210 2024-03-16T11:04:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.864+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400ccf0 con 0x7fc7cc107210 2024-03-16T11:04:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.864+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7cc1976c0 con 0x7fc7cc107210 2024-03-16T11:04:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.864+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7cc197bd0 con 0x7fc7cc107210 2024-03-16T11:04:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.864+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc7cc0629e0 con 0x7fc7cc107210 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.866+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc7c4005000 con 0x7fc7cc107210 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.866+0000 7fc7b3fff700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7b40a29a0 0x7fc7b40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.866+0000 7fc7caffd700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7b40a29a0 0x7fc7b40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.866+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fc7c40cefa0 con 0x7fc7cc107210 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.867+0000 7fc7b3fff700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] conn(0x7fc7b40ac180 0x7fc7b40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.867+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 --> [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7b40aec50 con 0x7fc7b40ac180 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.867+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fc7c40cf480 con 0x7fc7cc107210 2024-03-16T11:04:36.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.867+0000 7fc7caffd700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7b40a29a0 0x7fc7b40a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00f540 tx=0x7fc7bc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:36.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.867+0000 7fc7ca7fc700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] conn(0x7fc7b40ac180 0x7fc7b40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:36.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.867+0000 7fc7ca7fc700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] conn(0x7fc7b40ac180 0x7fc7b40ae5a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:36.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.868+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== osd.11 v2:172.21.15.153:6824/1068297198 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7b40aec50 con 0x7fc7b40ac180 2024-03-16T11:04:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.885+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 --> [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc7cc062de0 con 0x7fc7b40ac180 2024-03-16T11:04:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7b3fff700 1 -- 172.21.15.78:0/1639603624 <== osd.11 v2:172.21.15.153:6824/1068297198 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc7cc062de0 con 0x7fc7b40ac180 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] conn(0x7fc7b40ac180 msgr2=0x7fc7b40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] conn(0x7fc7b40ac180 0x7fc7b40ae5a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7b40a29a0 msgr2=0x7fc7b40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7b40a29a0 0x7fc7b40a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00f540 tx=0x7fc7bc00b040 comp rx=0 tx=0).stop 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc107210 msgr2=0x7fc7cc1a48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 --2- 172.21.15.78:0/1639603624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7cc107210 0x7fc7cc1a48f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400e940 tx=0x7fc7c400ec50 comp rx=0 tx=0).stop 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7cb7fe700 1 -- 172.21.15.78:0/1639603624 reap_dead start 2024-03-16T11:04:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 shutdown_connections 2024-03-16T11:04:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 >> 172.21.15.78:0/1639603624 conn(0x7fc7cc078100 msgr2=0x7fc7cc1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 shutdown_connections 2024-03-16T11:04:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.886+0000 7fc7d14da700 1 -- 172.21.15.78:0/1639603624 wait complete. 2024-03-16T11:04:37.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:37 smithi078 ceph-mon[32235]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T11:04:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:37 smithi098 ceph-mon[35848]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T11:04:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:37 smithi153 ceph-mon[35521]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T11:04:38.055 INFO:teuthology.orchestra.run.smithi078.stdout:34359738443 2024-03-16T11:04:38.055 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-16T11:04:38.590 INFO:teuthology.orchestra.run.smithi078.stdout:55834574918 2024-03-16T11:04:38.590 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-16T11:04:38.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 -- 172.21.15.78:0/2045439537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 msgr2=0x7efc3c101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 --2- 172.21.15.78:0/2045439537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 0x7efc3c101840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efc380099c0 tx=0x7efc38009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 -- 172.21.15.78:0/2045439537 shutdown_connections 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 --2- 172.21.15.78:0/2045439537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc3c1064b0 0x7efc3c10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 --2- 172.21.15.78:0/2045439537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 0x7efc3c101840 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 --2- 172.21.15.78:0/2045439537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc3c101e10 0x7efc3c105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 -- 172.21.15.78:0/2045439537 >> 172.21.15.78:0/2045439537 conn(0x7efc3c077220 msgr2=0x7efc3c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 -- 172.21.15.78:0/2045439537 shutdown_connections 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 -- 172.21.15.78:0/2045439537 wait complete. 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 Processor -- start 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.984+0000 7efc43445700 1 -- start start 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc43445700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 0x7efc3c1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc43445700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc3c101e10 0x7efc3c1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc43445700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc3c1064b0 0x7efc3c19aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc43445700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c114670 con 0x7efc3c101460 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc43445700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1144f0 con 0x7efc3c1064b0 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc43445700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1147f0 con 0x7efc3c101e10 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 0x7efc3c1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 0x7efc3c1a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38732/0 (socket says 172.21.15.78:38732) 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 -- 172.21.15.78:0/2257573885 learned_addr learned my addr 172.21.15.78:0/2257573885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc419e2700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc3c1064b0 0x7efc3c19aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc409e0700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc3c101e10 0x7efc3c1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 -- 172.21.15.78:0/2257573885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc3c101e10 msgr2=0x7efc3c1a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc3c101e10 0x7efc3c1a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 -- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc3c1064b0 msgr2=0x7efc3c19aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc3c1064b0 0x7efc3c19aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 -- 172.21.15.78:0/2257573885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c19b190 con 0x7efc3c101460 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc419e2700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc3c1064b0 0x7efc3c19aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc411e1700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 0x7efc3c1a0820 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efc38006010 tx=0x7efc38004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc38026070 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3c19b420 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc38004630 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.985+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3800fb20 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.986+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc3c1a76f0 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.986+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc3c062a60 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.987+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efc38029070 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc327fc700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efc280a2900 0x7efc280a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc409e0700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efc280a2900 0x7efc280a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7efc380cf920 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc327fc700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] conn(0x7efc280abfb0 0x7efc280ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 --> [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc280aea80 con 0x7efc280abfb0 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7efc380cfd80 con 0x7efc3c101460 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc419e2700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] conn(0x7efc280abfb0 0x7efc280ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.988+0000 7efc409e0700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efc280a2900 0x7efc280a4dc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efc3c100ee0 tx=0x7efc2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:38.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.989+0000 7efc419e2700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] conn(0x7efc280abfb0 0x7efc280ae3d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:38.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:38.989+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== osd.7 v2:172.21.15.98:6816/999107021 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efc280aea80 con 0x7efc280abfb0 2024-03-16T11:04:39.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.007+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 --> [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efc3c062e60 con 0x7efc280abfb0 2024-03-16T11:04:39.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.007+0000 7efc327fc700 1 -- 172.21.15.78:0/2257573885 <== osd.7 v2:172.21.15.98:6816/999107021 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efc3c062e60 con 0x7efc280abfb0 2024-03-16T11:04:39.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.007+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] conn(0x7efc280abfb0 msgr2=0x7efc280ae3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.007+0000 7efc43445700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] conn(0x7efc280abfb0 0x7efc280ae3d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efc280a2900 msgr2=0x7efc280a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efc280a2900 0x7efc280a4dc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efc3c100ee0 tx=0x7efc2c009260 comp rx=0 tx=0).stop 2024-03-16T11:04:39.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 msgr2=0x7efc3c1a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 --2- 172.21.15.78:0/2257573885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc3c101460 0x7efc3c1a0820 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efc38006010 tx=0x7efc38004270 comp rx=0 tx=0).stop 2024-03-16T11:04:39.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc419e2700 1 -- 172.21.15.78:0/2257573885 reap_dead start 2024-03-16T11:04:39.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 shutdown_connections 2024-03-16T11:04:39.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 >> 172.21.15.78:0/2257573885 conn(0x7efc3c077220 msgr2=0x7efc3c105190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:39.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 shutdown_connections 2024-03-16T11:04:39.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.008+0000 7efc43445700 1 -- 172.21.15.78:0/2257573885 wait complete. 2024-03-16T11:04:39.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:39 smithi078 ceph-mon[32235]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 -- 172.21.15.78:0/4198511766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e006a4c0 msgr2=0x7f00e006a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 --2- 172.21.15.78:0/4198511766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e006a4c0 0x7f00e006a920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009990 tx=0x7f00d0009ca0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 -- 172.21.15.78:0/4198511766 shutdown_connections 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 --2- 172.21.15.78:0/4198511766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00e01071c0 0x7f00e0069f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 --2- 172.21.15.78:0/4198511766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00e0104cd0 0x7f00e0105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 --2- 172.21.15.78:0/4198511766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e006a4c0 0x7f00e006a920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 -- 172.21.15.78:0/4198511766 >> 172.21.15.78:0/4198511766 conn(0x7f00e0078100 msgr2=0x7f00e0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 -- 172.21.15.78:0/4198511766 shutdown_connections 2024-03-16T11:04:39.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.215+0000 7f00e84db700 1 -- 172.21.15.78:0/4198511766 wait complete. 2024-03-16T11:04:39.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.216+0000 7f00e84db700 1 Processor -- start 2024-03-16T11:04:39.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.216+0000 7f00e84db700 1 -- start start 2024-03-16T11:04:39.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.216+0000 7f00e84db700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00e006a4c0 0x7f00e019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:39.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.216+0000 7f00e84db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00e0104cd0 0x7f00e019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.216+0000 7f00e84db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e01071c0 0x7f00e01a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e84db700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f00e0118a20 con 0x7f00e0104cd0 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e84db700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00e01188a0 con 0x7f00e01071c0 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00e006a4c0 0x7f00e019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e5a76700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00e0104cd0 0x7f00e019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00e006a4c0 0x7f00e019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:42466/0 (socket says 172.21.15.78:42466) 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 -- 172.21.15.78:0/856482407 learned_addr learned my addr 172.21.15.78:0/856482407 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6a78700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e01071c0 0x7f00e01a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00e0118ba0 con 0x7f00e006a4c0 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 -- 172.21.15.78:0/856482407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e01071c0 msgr2=0x7f00e01a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e01071c0 0x7f00e01a3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 -- 172.21.15.78:0/856482407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00e0104cd0 msgr2=0x7f00e019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00e0104cd0 0x7f00e019f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 -- 172.21.15.78:0/856482407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e01a3fa0 con 0x7f00e006a4c0 2024-03-16T11:04:39.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6a78700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e01071c0 0x7f00e01a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:39.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e6277700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00e006a4c0 0x7f00e019ef90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00e9a0 tx=0x7f00dc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:39.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc00c9c0 con 0x7f00e006a4c0 2024-03-16T11:04:39.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.217+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e01a4290 con 0x7f00e006a4c0 2024-03-16T11:04:39.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.218+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00dc00cb60 con 0x7f00e006a4c0 2024-03-16T11:04:39.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.218+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e01abb10 con 0x7f00e006a4c0 2024-03-16T11:04:39.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.218+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc018680 con 0x7f00e006a4c0 2024-03-16T11:04:39.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.219+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f00c4001060 con 0x7f00e006a4c0 2024-03-16T11:04:39.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.219+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f00dc00cd10 con 0x7f00e006a4c0 2024-03-16T11:04:39.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.220+0000 7f00d77fe700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f00cc0a29a0 0x7f00cc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.220+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f00dc0cea10 con 0x7f00e006a4c0 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.220+0000 7f00e5a76700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f00cc0a29a0 0x7f00cc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.220+0000 7f00d77fe700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] conn(0x7f00cc0ac180 0x7f00cc0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.220+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 --> [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00dc01ce70 con 0x7f00cc0ac180 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.220+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f00dc0d32c0 con 0x7f00e006a4c0 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.221+0000 7f00e6a78700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] conn(0x7f00cc0ac180 0x7f00cc0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.221+0000 7f00e5a76700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f00cc0a29a0 0x7f00cc0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f00d000b2e0 tx=0x7f00d0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.221+0000 7f00e6a78700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] conn(0x7f00cc0ac180 0x7f00cc0ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:39.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.222+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== osd.5 v2:172.21.15.153:6808/1494126150 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00dc01ce70 con 0x7f00cc0ac180 2024-03-16T11:04:39.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.238+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 --> [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f00c4002e90 con 0x7f00cc0ac180 2024-03-16T11:04:39.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00d77fe700 1 -- 172.21.15.78:0/856482407 <== osd.5 v2:172.21.15.153:6808/1494126150 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f00c4002e90 con 0x7f00cc0ac180 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] conn(0x7f00cc0ac180 msgr2=0x7f00cc0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] conn(0x7f00cc0ac180 0x7f00cc0ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f00cc0a29a0 msgr2=0x7f00cc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f00cc0a29a0 0x7f00cc0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f00d000b2e0 tx=0x7f00d0005960 comp rx=0 tx=0).stop 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00e006a4c0 msgr2=0x7f00e019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 --2- 172.21.15.78:0/856482407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00e006a4c0 0x7f00e019ef90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00e9a0 tx=0x7f00dc00ecb0 comp rx=0 tx=0).stop 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e6a78700 1 -- 172.21.15.78:0/856482407 reap_dead start 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 shutdown_connections 2024-03-16T11:04:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 >> 172.21.15.78:0/856482407 conn(0x7f00e0078100 msgr2=0x7f00e0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 shutdown_connections 2024-03-16T11:04:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:39.239+0000 7f00e84db700 1 -- 172.21.15.78:0/856482407 wait complete. 2024-03-16T11:04:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:39 smithi098 ceph-mon[35848]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-16T11:04:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:39 smithi153 ceph-mon[35521]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-16T11:04:39.946 INFO:teuthology.orchestra.run.smithi078.stdout:111669149757 2024-03-16T11:04:39.946 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-16T11:04:41.019 INFO:teuthology.orchestra.run.smithi078.stdout:227633266729 2024-03-16T11:04:41.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-16T11:04:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 -- 172.21.15.78:0/3142793603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc102f30 msgr2=0x7f81cc106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 --2- 172.21.15.78:0/3142793603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc102f30 0x7f81cc106f70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0099c0 tx=0x7f81bc009cd0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 -- 172.21.15.78:0/3142793603 shutdown_connections 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 --2- 172.21.15.78:0/3142793603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81cc102580 0x7f81cc102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 --2- 172.21.15.78:0/3142793603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc107650 0x7f81cc10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 --2- 172.21.15.78:0/3142793603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc102f30 0x7f81cc106f70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 -- 172.21.15.78:0/3142793603 >> 172.21.15.78:0/3142793603 conn(0x7f81cc079430 msgr2=0x7f81cc079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 -- 172.21.15.78:0/3142793603 shutdown_connections 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.759+0000 7f81d3b2c700 1 -- 172.21.15.78:0/3142793603 wait complete. 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.760+0000 7f81d3b2c700 1 Processor -- start 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.760+0000 7f81d3b2c700 1 -- start start 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d3b2c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc102580 0x7f81cc19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d3b2c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81cc102f30 0x7f81cc19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d3b2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc107650 0x7f81cc1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d3b2c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1124e0 con 0x7f81cc107650 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d3b2c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc112360 con 0x7f81cc102580 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d3b2c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc112660 con 0x7f81cc102f30 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc107650 0x7f81cc1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d10c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81cc102f30 0x7f81cc19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc107650 0x7f81cc1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38696/0 (socket says 172.21.15.78:38696) 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 -- 172.21.15.78:0/1547237024 learned_addr learned my addr 172.21.15.78:0/1547237024 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 -- 172.21.15.78:0/1547237024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81cc102f30 msgr2=0x7f81cc19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81cc102f30 0x7f81cc19ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 -- 172.21.15.78:0/1547237024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc102580 msgr2=0x7f81cc19e720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc102580 0x7f81cc19e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 -- 172.21.15.78:0/1547237024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc199040 con 0x7f81cc107650 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d10c7700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81cc102f30 0x7f81cc19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.761+0000 7f81d20c9700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc107650 0x7f81cc1988e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f81c400efc0 tx=0x7f81c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.762+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c40092e0 con 0x7f81cc107650 2024-03-16T11:04:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.762+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81cc1992d0 con 0x7f81cc107650 2024-03-16T11:04:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.762+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81c4004bb0 con 0x7f81cc107650 2024-03-16T11:04:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.762+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4004020 con 0x7f81cc107650 2024-03-16T11:04:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.762+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81cc1a55f0 con 0x7f81cc107650 2024-03-16T11:04:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.763+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81cc0629e0 con 0x7f81cc107650 2024-03-16T11:04:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.764+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f81c401b030 con 0x7f81cc107650 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.764+0000 7f81c2ffd700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81b80a2920 0x7f81b80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.764+0000 7f81d18c8700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81b80a2920 0x7f81b80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.764+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f81c40cf050 con 0x7f81cc107650 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.765+0000 7f81c2ffd700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] conn(0x7f81b80ac100 0x7f81b80ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.765+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 --> [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81b80aebd0 con 0x7f81b80ac100 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.765+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f81c40cf530 con 0x7f81cc107650 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.765+0000 7f81d10c7700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] conn(0x7f81b80ac100 0x7f81b80ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.765+0000 7f81d18c8700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81b80a2920 0x7f81b80a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81cc199a20 tx=0x7f81c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.765+0000 7f81d10c7700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] conn(0x7f81b80ac100 0x7f81b80ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.766+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== osd.6 v2:172.21.15.78:6818/1730932549 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81b80aebd0 con 0x7f81b80ac100 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.782+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 --> [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f81cc062de0 con 0x7f81b80ac100 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81c2ffd700 1 -- 172.21.15.78:0/1547237024 <== osd.6 v2:172.21.15.78:6818/1730932549 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f81cc062de0 con 0x7f81b80ac100 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] conn(0x7f81b80ac100 msgr2=0x7f81b80ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:41.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] conn(0x7f81b80ac100 0x7f81b80ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81b80a2920 msgr2=0x7f81b80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81b80a2920 0x7f81b80a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81cc199a20 tx=0x7f81c8009260 comp rx=0 tx=0).stop 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc107650 msgr2=0x7f81cc1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 --2- 172.21.15.78:0/1547237024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc107650 0x7f81cc1988e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f81c400efc0 tx=0x7f81c400c3a0 comp rx=0 tx=0).stop 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d20c9700 1 -- 172.21.15.78:0/1547237024 reap_dead start 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 shutdown_connections 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 >> 172.21.15.78:0/1547237024 conn(0x7f81cc079430 msgr2=0x7f81cc100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 shutdown_connections 2024-03-16T11:04:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:36.783+0000 7f81d3b2c700 1 -- 172.21.15.78:0/1547237024 wait complete. 2024-03-16T11:04:41.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:41 smithi078 ceph-mon[32235]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T11:04:41.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:41 smithi098 ceph-mon[35848]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T11:04:41.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:41 smithi153 ceph-mon[35521]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T11:04:41.689 INFO:teuthology.orchestra.run.smithi078.stdout:206158430254 2024-03-16T11:04:41.689 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-16T11:04:42.376 INFO:teuthology.orchestra.run.smithi078.stdout:171798691892 2024-03-16T11:04:42.377 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-16T11:04:42.495 INFO:teuthology.orchestra.run.smithi078.stdout:77309411393 2024-03-16T11:04:42.496 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-16T11:04:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 -- 172.21.15.78:0/3299908628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c06b3e0 msgr2=0x7f6e0c10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 --2- 172.21.15.78:0/3299908628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c06b3e0 0x7f6e0c10f120 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400b0d0 tx=0x7f6e0400b3e0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 -- 172.21.15.78:0/3299908628 shutdown_connections 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 --2- 172.21.15.78:0/3299908628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e0c108300 0x7f6e0c1086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 --2- 172.21.15.78:0/3299908628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c06b3e0 0x7f6e0c10f120 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 --2- 172.21.15.78:0/3299908628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e0c06aa20 0x7f6e0c06aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 -- 172.21.15.78:0/3299908628 >> 172.21.15.78:0/3299908628 conn(0x7f6e0c0781e0 msgr2=0x7f6e0c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.667+0000 7f6e135a1700 1 -- 172.21.15.78:0/3299908628 shutdown_connections 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 -- 172.21.15.78:0/3299908628 wait complete. 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 Processor -- start 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 -- start start 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e0c06aa20 0x7f6e0c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e0c06b3e0 0x7f6e0c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c108300 0x7f6e0c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c112430 con 0x7f6e0c108300 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c1122b0 con 0x7f6e0c06aa20 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e135a1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c1125b0 con 0x7f6e0c06b3e0 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.668+0000 7f6e11b3e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c108300 0x7f6e0c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c108300 0x7f6e0c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38772/0 (socket says 172.21.15.78:38772) 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 -- 172.21.15.78:0/1589202786 learned_addr learned my addr 172.21.15.78:0/1589202786 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e1133d700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e0c06aa20 0x7f6e0c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e10b3c700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e0c06b3e0 0x7f6e0c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 -- 172.21.15.78:0/1589202786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e0c06b3e0 msgr2=0x7f6e0c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e0c06b3e0 0x7f6e0c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 -- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e0c06aa20 msgr2=0x7f6e0c19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e0c06aa20 0x7f6e0c19e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 -- 172.21.15.78:0/1589202786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0c198f90 con 0x7f6e0c108300 2024-03-16T11:04:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e11b3e700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c108300 0x7f6e0c1986b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009850 tx=0x7f6e04007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0401f070 con 0x7f6e0c108300 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e04004c50 con 0x7f6e0c108300 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0401b670 con 0x7f6e0c108300 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0c199220 con 0x7f6e0c108300 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.669+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e0c1a5560 con 0x7f6e0c108300 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.670+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e0c062a20 con 0x7f6e0c108300 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.671+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6e04022070 con 0x7f6e0c108300 2024-03-16T11:04:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e027fc700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6df80a29a0 0x7f6df80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e1133d700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6df80a29a0 0x7f6df80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f6e0400f020 con 0x7f6e0c108300 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e027fc700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] conn(0x7f6df80ac180 0x7f6df80ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 --> [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6df80aec50 con 0x7f6df80ac180 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f6e040d0200 con 0x7f6e0c108300 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e1133d700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6df80a29a0 0x7f6df80a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e08007be0 tx=0x7f6e080073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.672+0000 7f6e10b3c700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] conn(0x7f6df80ac180 0x7f6df80ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.673+0000 7f6e10b3c700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] conn(0x7f6df80ac180 0x7f6df80ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.673+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== osd.4 v2:172.21.15.98:6808/2809778583 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6df80aec50 con 0x7f6df80ac180 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.689+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 --> [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6e0c062e20 con 0x7f6df80ac180 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e027fc700 1 -- 172.21.15.78:0/1589202786 <== osd.4 v2:172.21.15.98:6808/2809778583 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6e0c062e20 con 0x7f6df80ac180 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] conn(0x7f6df80ac180 msgr2=0x7f6df80ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] conn(0x7f6df80ac180 0x7f6df80ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6df80a29a0 msgr2=0x7f6df80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6df80a29a0 0x7f6df80a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e08007be0 tx=0x7f6e080073d0 comp rx=0 tx=0).stop 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c108300 msgr2=0x7f6e0c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 --2- 172.21.15.78:0/1589202786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e0c108300 0x7f6e0c1986b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009850 tx=0x7f6e04007a20 comp rx=0 tx=0).stop 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e11b3e700 1 -- 172.21.15.78:0/1589202786 reap_dead start 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 shutdown_connections 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 >> 172.21.15.78:0/1589202786 conn(0x7f6e0c0781e0 msgr2=0x7f6e0c0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 shutdown_connections 2024-03-16T11:04:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:40.690+0000 7f6e135a1700 1 -- 172.21.15.78:0/1589202786 wait complete. 2024-03-16T11:04:43.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:43 smithi153 ceph-mon[35521]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-16T11:04:43.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:43 smithi078 ceph-mon[32235]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-16T11:04:43.415 INFO:teuthology.orchestra.run.smithi078.stdout:146028888119 2024-03-16T11:04:43.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-16T11:04:43.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:43 smithi098 ceph-mon[35848]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-16T11:04:43.816 INFO:teuthology.orchestra.run.smithi078.stdout:356482285588 2024-03-16T11:04:43.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-16T11:04:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:44.359 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:44.360 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:45.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:45 smithi078 ceph-mon[32235]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:04:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:45 smithi098 ceph-mon[35848]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:04:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:45 smithi153 ceph-mon[35521]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:04:47.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:47 smithi078 ceph-mon[32235]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:04:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:47 smithi098 ceph-mon[35848]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:04:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:47 smithi153 ceph-mon[35521]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 -- 172.21.15.78:0/3620437300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8241064b0 msgr2=0x7fa82410e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 --2- 172.21.15.78:0/3620437300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8241064b0 0x7fa82410e080 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00b0d0 tx=0x7fa81c00b3e0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 -- 172.21.15.78:0/3620437300 shutdown_connections 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 --2- 172.21.15.78:0/3620437300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8241064b0 0x7fa82410e080 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 --2- 172.21.15.78:0/3620437300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa824101460 0x7fa824101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 --2- 172.21.15.78:0/3620437300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa824101e10 0x7fa824105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 -- 172.21.15.78:0/3620437300 >> 172.21.15.78:0/3620437300 conn(0x7fa824077220 msgr2=0x7fa824077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:47.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 -- 172.21.15.78:0/3620437300 shutdown_connections 2024-03-16T11:04:47.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.631+0000 7fa82bbb4700 1 -- 172.21.15.78:0/3620437300 wait complete. 2024-03-16T11:04:47.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 Processor -- start 2024-03-16T11:04:47.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 -- start start 2024-03-16T11:04:47.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa824101460 0x7fa824199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:47.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa824101e10 0x7fa824199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:47.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8241064b0 0x7fa8241a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:47.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa824111370 con 0x7fa824101460 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241111f0 con 0x7fa8241064b0 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82bbb4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241114f0 con 0x7fa824101e10 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa824101460 0x7fa824199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82914f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa824101e10 0x7fa824199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa824101460 0x7fa824199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56902/0 (socket says 172.21.15.78:56902) 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82914f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa824101e10 0x7fa824199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57986/0 (socket says 172.21.15.78:57986) 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 -- 172.21.15.78:0/2762952908 learned_addr learned my addr 172.21.15.78:0/2762952908 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82a151700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8241064b0 0x7fa8241a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 -- 172.21.15.78:0/2762952908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa824101e10 msgr2=0x7fa824199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa824101e10 0x7fa824199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 -- 172.21.15.78:0/2762952908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8241064b0 msgr2=0x7fa8241a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8241064b0 0x7fa8241a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa829950700 1 -- 172.21.15.78:0/2762952908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8241a4d30 con 0x7fa824101460 2024-03-16T11:04:47.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82914f700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa824101e10 0x7fa824199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.632+0000 7fa82a151700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8241064b0 0x7fa8241a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.633+0000 7fa829950700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa824101460 0x7fa824199770 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e9a0 tx=0x7fa82000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.633+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82000ca10 con 0x7fa824101460 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.633+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa82000cbb0 con 0x7fa824101460 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.633+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8241a4fc0 con 0x7fa824101460 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.633+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820018680 con 0x7fa824101460 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.633+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8241a5470 con 0x7fa824101460 2024-03-16T11:04:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.633+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa8240629a0 con 0x7fa824101460 2024-03-16T11:04:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.635+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa820010430 con 0x7fa824101460 2024-03-16T11:04:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.635+0000 7fa81affd700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8100a2920 0x7fa8100a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa82914f700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8100a2920 0x7fa8100a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fa8200cf090 con 0x7fa824101460 2024-03-16T11:04:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa81affd700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] conn(0x7fa8100ac100 0x7fa8100ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 --> [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa82000ae50 con 0x7fa8100ac100 2024-03-16T11:04:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fa8200cf570 con 0x7fa824101460 2024-03-16T11:04:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa82a151700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] conn(0x7fa8100ac100 0x7fa8100ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa82914f700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8100a2920 0x7fa8100a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa824100d30 tx=0x7fa814009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.636+0000 7fa82a151700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] conn(0x7fa8100ac100 0x7fa8100ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.637+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== osd.9 v2:172.21.15.78:6826/2205607708 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa82000ae50 con 0x7fa8100ac100 2024-03-16T11:04:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.653+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 --> [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa824062dd0 con 0x7fa8100ac100 2024-03-16T11:04:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.653+0000 7fa81affd700 1 -- 172.21.15.78:0/2762952908 <== osd.9 v2:172.21.15.78:6826/2205607708 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa824062dd0 con 0x7fa8100ac100 2024-03-16T11:04:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.653+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] conn(0x7fa8100ac100 msgr2=0x7fa8100ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.653+0000 7fa82bbb4700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] conn(0x7fa8100ac100 0x7fa8100ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8100a2920 msgr2=0x7fa8100a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8100a2920 0x7fa8100a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa824100d30 tx=0x7fa814009260 comp rx=0 tx=0).stop 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa824101460 msgr2=0x7fa824199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 --2- 172.21.15.78:0/2762952908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa824101460 0x7fa824199770 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e9a0 tx=0x7fa82000ecb0 comp rx=0 tx=0).stop 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82a151700 1 -- 172.21.15.78:0/2762952908 reap_dead start 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 shutdown_connections 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 >> 172.21.15.78:0/2762952908 conn(0x7fa824077220 msgr2=0x7fa8240ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 shutdown_connections 2024-03-16T11:04:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:47.654+0000 7fa82bbb4700 1 -- 172.21.15.78:0/2762952908 wait complete. 2024-03-16T11:04:49.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:49 smithi078 ceph-mon[32235]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:04:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:49 smithi098 ceph-mon[35848]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:04:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:49 smithi153 ceph-mon[35521]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:04:51.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:51 smithi078 ceph-mon[32235]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:04:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:51 smithi098 ceph-mon[35848]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:04:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:51 smithi153 ceph-mon[35521]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.942+0000 7f2e84638700 1 -- 172.21.15.78:0/1216087145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 msgr2=0x7f2e7c107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.942+0000 7f2e84638700 1 --2- 172.21.15.78:0/1216087145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c107640 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009990 tx=0x7f2e6c009ca0 comp rx=0 tx=0).stop 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- 172.21.15.78:0/1216087145 shutdown_connections 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 --2- 172.21.15.78:0/1216087145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e7c101b70 0x7f2e7c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 --2- 172.21.15.78:0/1216087145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c107640 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 --2- 172.21.15.78:0/1216087145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e7c1011b0 0x7f2e7c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- 172.21.15.78:0/1216087145 >> 172.21.15.78:0/1216087145 conn(0x7f2e7c0781e0 msgr2=0x7f2e7c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- 172.21.15.78:0/1216087145 shutdown_connections 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- 172.21.15.78:0/1216087145 wait complete. 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 Processor -- start 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- start start 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e7c1011b0 0x7f2e7c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e7c101b70 0x7f2e7c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c111350 con 0x7f2e7c107260 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1111d0 con 0x7f2e7c1011b0 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.943+0000 7f2e84638700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1114d0 con 0x7f2e7c101b70 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e823d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e7c1011b0 0x7f2e7c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56928/0 (socket says 172.21.15.78:56928) 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 -- 172.21.15.78:0/1558484365 learned_addr learned my addr 172.21.15.78:0/1558484365 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e81bd3700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e7c101b70 0x7f2e7c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 -- 172.21.15.78:0/1558484365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e7c101b70 msgr2=0x7f2e7c19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e7c101b70 0x7f2e7c19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 -- 172.21.15.78:0/1558484365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e7c1011b0 msgr2=0x7f2e7c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e7c1011b0 0x7f2e7c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 -- 172.21.15.78:0/1558484365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7c199010 con 0x7f2e7c107260 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e82bd5700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c198730 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400ef30 tx=0x7f2e7400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e737fe700 1 -- 172.21.15.78:0/1558484365 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e740092e0 con 0x7f2e7c107260 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e737fe700 1 -- 172.21.15.78:0/1558484365 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e74004bb0 con 0x7f2e7c107260 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7c1992a0 con 0x7f2e7c107260 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e737fe700 1 -- 172.21.15.78:0/1558484365 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74004020 con 0x7f2e7c107260 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.944+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7c1a55e0 con 0x7f2e7c107260 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.945+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e7c199610 con 0x7f2e7c107260 2024-03-16T11:04:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.946+0000 7f2e737fe700 1 -- 172.21.15.78:0/1558484365 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2e7401b030 con 0x7f2e7c107260 2024-03-16T11:04:51.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.947+0000 7f2e737fe700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2e680a29a0 0x7f2e680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:51.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.947+0000 7f2e823d4700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2e680a29a0 0x7f2e680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:51.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.947+0000 7f2e737fe700 1 -- 172.21.15.78:0/1558484365 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f2e740cee20 con 0x7f2e7c107260 2024-03-16T11:04:51.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.947+0000 7f2e823d4700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2e680a29a0 0x7f2e680a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00b2e0 tx=0x7f2e6c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:51.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:51.951+0000 7f2e737fe700 1 -- 172.21.15.78:0/1558484365 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e7c199610 con 0x7f2e7c107260 2024-03-16T11:04:51.977 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:51.978 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:04:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.159+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2e7c1a57b0 con 0x7f2e7c107260 2024-03-16T11:04:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.159+0000 7f2e737fe700 1 -- 172.21.15.78:0/1558484365 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2e74022020 con 0x7f2e7c107260 2024-03-16T11:04:52.159 INFO:teuthology.orchestra.run.smithi078.stdout:111669149760 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2e680a29a0 msgr2=0x7f2e680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2e680a29a0 0x7f2e680a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00b2e0 tx=0x7f2e6c005960 comp rx=0 tx=0).stop 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 msgr2=0x7f2e7c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c198730 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400ef30 tx=0x7f2e7400c3a0 comp rx=0 tx=0).stop 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 shutdown_connections 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2e680a29a0 0x7f2e680a4e60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e7c107260 0x7f2e7c198730 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e7c1011b0 0x7f2e7c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 --2- 172.21.15.78:0/1558484365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e7c101b70 0x7f2e7c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 >> 172.21.15.78:0/1558484365 conn(0x7f2e7c0781e0 msgr2=0x7f2e7c1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:52.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 shutdown_connections 2024-03-16T11:04:52.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.161+0000 7f2e84638700 1 -- 172.21.15.78:0/1558484365 wait complete. 2024-03-16T11:04:52.899 INFO:teuthology.orchestra.run.smithi078.stdout:296352743458 2024-03-16T11:04:52.900 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-16T11:04:53.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:53 smithi078 ceph-mon[32235]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:04:53.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:53 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1558484365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T11:04:53.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:53 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/156466223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T11:04:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:53 smithi098 ceph-mon[35848]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:04:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:53 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1558484365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T11:04:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:53 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/156466223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T11:04:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:53 smithi153 ceph-mon[35521]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:04:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:53 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1558484365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T11:04:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:53 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/156466223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T11:04:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 -- 172.21.15.78:0/177997102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc1064f0 msgr2=0x7f47bc10e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 --2- 172.21.15.78:0/177997102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc1064f0 0x7f47bc10e0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f47b400b0a0 tx=0x7f47b400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 -- 172.21.15.78:0/177997102 shutdown_connections 2024-03-16T11:04:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 --2- 172.21.15.78:0/177997102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc1064f0 0x7f47bc10e0c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 --2- 172.21.15.78:0/177997102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc101460 0x7f47bc101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 --2- 172.21.15.78:0/177997102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc101e10 0x7f47bc105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 -- 172.21.15.78:0/177997102 >> 172.21.15.78:0/177997102 conn(0x7f47bc0771f0 msgr2=0x7f47bc077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 -- 172.21.15.78:0/177997102 shutdown_connections 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.589+0000 7f47c270d700 1 -- 172.21.15.78:0/177997102 wait complete. 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 Processor -- start 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 -- start start 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc101460 0x7f47bc19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc101e10 0x7f47bc19db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc1064f0 0x7f47bc197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc1113d0 con 0x7f47bc101460 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc111250 con 0x7f47bc101e10 2024-03-16T11:04:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c270d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc111550 con 0x7f47bc1064f0 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc101460 0x7f47bc19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc101460 0x7f47bc19d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56938/0 (socket says 172.21.15.78:56938) 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 -- 172.21.15.78:0/156466223 learned_addr learned my addr 172.21.15.78:0/156466223 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bb7fe700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc101e10 0x7f47bc19db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47c0caa700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc1064f0 0x7f47bc197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 -- 172.21.15.78:0/156466223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc1064f0 msgr2=0x7f47bc197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc1064f0 0x7f47bc197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 -- 172.21.15.78:0/156466223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc101e10 msgr2=0x7f47bc19db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc101e10 0x7f47bc19db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.590+0000 7f47bbfff700 1 -- 172.21.15.78:0/156466223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47bc197ff0 con 0x7f47bc101460 2024-03-16T11:04:54.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.591+0000 7f47bbfff700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc101460 0x7f47bc19d5f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00d980 tx=0x7f47ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:54.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.591+0000 7f47b97fa700 1 -- 172.21.15.78:0/156466223 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac016070 con 0x7f47bc101460 2024-03-16T11:04:54.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.591+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47bc1982e0 con 0x7f47bc101460 2024-03-16T11:04:54.823 INFO:teuthology.orchestra.run.smithi078.stdout:322122547229 2024-03-16T11:04:54.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.591+0000 7f47b97fa700 1 -- 172.21.15.78:0/156466223 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47ac004830 con 0x7f47bc101460 2024-03-16T11:04:54.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.591+0000 7f47b97fa700 1 -- 172.21.15.78:0/156466223 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac005530 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.591+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47bc1a4570 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.592+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47bc1028f0 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.593+0000 7f47b97fa700 1 -- 172.21.15.78:0/156466223 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f47ac0049e0 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.593+0000 7f47b97fa700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f47a80a2920 0x7f47a80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.594+0000 7f47bb7fe700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f47a80a2920 0x7f47a80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.594+0000 7f47b97fa700 1 -- 172.21.15.78:0/156466223 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f47ac0ceca0 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.594+0000 7f47bb7fe700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f47a80a2920 0x7f47a80a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f47bc198e80 tx=0x7f47a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.597+0000 7f47b97fa700 1 -- 172.21.15.78:0/156466223 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f47ac01f020 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.799+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f47bc198bf0 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.799+0000 7f47b97fa700 1 -- 172.21.15.78:0/156466223 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f47ac09bd90 con 0x7f47bc101460 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f47a80a2920 msgr2=0x7f47a80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f47a80a2920 0x7f47a80a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f47bc198e80 tx=0x7f47a400b410 comp rx=0 tx=0).stop 2024-03-16T11:04:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc101460 msgr2=0x7f47bc19d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc101460 0x7f47bc19d5f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00d980 tx=0x7f47ac00dc90 comp rx=0 tx=0).stop 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 shutdown_connections 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f47a80a2920 0x7f47a80a4de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc1064f0 0x7f47bc197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47bc101460 0x7f47bc19d5f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 --2- 172.21.15.78:0/156466223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc101e10 0x7f47bc19db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 >> 172.21.15.78:0/156466223 conn(0x7f47bc0771f0 msgr2=0x7f47bc1009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 shutdown_connections 2024-03-16T11:04:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:52.802+0000 7f47c270d700 1 -- 172.21.15.78:0/156466223 wait complete. 2024-03-16T11:04:55.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:55 smithi078 ceph-mon[32235]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:04:55.418 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547223 got 322122547229 for osd.10 2024-03-16T11:04:55.419 DEBUG:teuthology.parallel:result is None 2024-03-16T11:04:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:55 smithi098 ceph-mon[35848]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:04:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:55 smithi153 ceph-mon[35521]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:04:55.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.889+0000 7f7d389a4700 1 -- 172.21.15.78:0/3329490861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d341011d0 msgr2=0x7f7d34101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:55.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.889+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3329490861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d341011d0 0x7f7d34101630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009990 tx=0x7f7d28009ca0 comp rx=0 tx=0).stop 2024-03-16T11:04:55.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.889+0000 7f7d389a4700 1 -- 172.21.15.78:0/3329490861 shutdown_connections 2024-03-16T11:04:55.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.889+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3329490861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d34107210 0x7f7d341075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:55.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.889+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3329490861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d34101b70 0x7f7d3410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:55.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.889+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3329490861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d341011d0 0x7f7d34101630 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:55.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.889+0000 7f7d389a4700 1 -- 172.21.15.78:0/3329490861 >> 172.21.15.78:0/3329490861 conn(0x7f7d34078100 msgr2=0x7f7d34078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.890+0000 7f7d389a4700 1 -- 172.21.15.78:0/3329490861 shutdown_connections 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.890+0000 7f7d389a4700 1 -- 172.21.15.78:0/3329490861 wait complete. 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.890+0000 7f7d389a4700 1 Processor -- start 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.890+0000 7f7d389a4700 1 -- start start 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.890+0000 7f7d389a4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d341011d0 0x7f7d3419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d389a4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d34101b70 0x7f7d3419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d389a4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d34107210 0x7f7d341987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d389a4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341124a0 con 0x7f7d34107210 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d389a4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d34112320 con 0x7f7d341011d0 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d389a4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d34112620 con 0x7f7d34101b70 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d34107210 0x7f7d341987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d34107210 0x7f7d341987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45452/0 (socket says 172.21.15.78:45452) 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 -- 172.21.15.78:0/3182756180 learned_addr learned my addr 172.21.15.78:0/3182756180 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d3259c700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d341011d0 0x7f7d3419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:55.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d31d9b700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d34101b70 0x7f7d3419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 -- 172.21.15.78:0/3182756180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d34101b70 msgr2=0x7f7d3419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d34101b70 0x7f7d3419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 -- 172.21.15.78:0/3182756180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d341011d0 msgr2=0x7f7d3419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d341011d0 0x7f7d3419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 -- 172.21.15.78:0/3182756180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d34198fc0 con 0x7f7d34107210 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d32d9d700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d34107210 0x7f7d341987e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c007ee0 tx=0x7f7d2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d237fe700 1 -- 172.21.15.78:0/3182756180 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00dd20 con 0x7f7d34107210 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d237fe700 1 -- 172.21.15.78:0/3182756180 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d2c004020 con 0x7f7d34107210 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d341992b0 con 0x7f7d34107210 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d237fe700 1 -- 172.21.15.78:0/3182756180 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c015660 con 0x7f7d34107210 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.891+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d341a54c0 con 0x7f7d34107210 2024-03-16T11:04:55.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.892+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d341028c0 con 0x7f7d34107210 2024-03-16T11:04:55.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.893+0000 7f7d237fe700 1 -- 172.21.15.78:0/3182756180 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7d2c0041d0 con 0x7f7d34107210 2024-03-16T11:04:55.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.894+0000 7f7d237fe700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7d1c0a2920 0x7f7d1c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:55.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.894+0000 7f7d3259c700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7d1c0a2920 0x7f7d1c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:55.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.894+0000 7f7d237fe700 1 -- 172.21.15.78:0/3182756180 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f7d2c0ce3f0 con 0x7f7d34107210 2024-03-16T11:04:55.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.894+0000 7f7d3259c700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7d1c0a2920 0x7f7d1c0a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009be0 tx=0x7f7d24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:55.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:55.899+0000 7f7d237fe700 1 -- 172.21.15.78:0/3182756180 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d2c010070 con 0x7f7d34107210 2024-03-16T11:04:56.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.100+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f7d3402db70 con 0x7f7d34107210 2024-03-16T11:04:56.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.101+0000 7f7d237fe700 1 -- 172.21.15.78:0/3182756180 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7d2c09b4e0 con 0x7f7d34107210 2024-03-16T11:04:56.101 INFO:teuthology.orchestra.run.smithi078.stdout:206158430258 2024-03-16T11:04:56.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7d1c0a2920 msgr2=0x7f7d1c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7d1c0a2920 0x7f7d1c0a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009be0 tx=0x7f7d24009260 comp rx=0 tx=0).stop 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d34107210 msgr2=0x7f7d341987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d34107210 0x7f7d341987e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c007ee0 tx=0x7f7d2c00d3b0 comp rx=0 tx=0).stop 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 shutdown_connections 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7d1c0a2920 0x7f7d1c0a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d34107210 0x7f7d341987e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d341011d0 0x7f7d3419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 --2- 172.21.15.78:0/3182756180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d34101b70 0x7f7d3419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 >> 172.21.15.78:0/3182756180 conn(0x7f7d34078100 msgr2=0x7f7d341000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 shutdown_connections 2024-03-16T11:04:56.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:56.103+0000 7f7d389a4700 1 -- 172.21.15.78:0/3182756180 wait complete. 2024-03-16T11:04:57.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430254 got 206158430258 for osd.6 2024-03-16T11:04:57.189 DEBUG:teuthology.parallel:result is None 2024-03-16T11:04:57.360 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:57 smithi078 ceph-mon[32235]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:04:57.360 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:57 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3182756180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T11:04:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:57 smithi098 ceph-mon[35848]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:04:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:57 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3182756180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T11:04:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:57 smithi153 ceph-mon[35521]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:04:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:57 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3182756180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T11:04:58.691 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149757 got 111669149760 for osd.3 2024-03-16T11:04:58.691 DEBUG:teuthology.parallel:result is None 2024-03-16T11:04:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:59 smithi078 ceph-mon[32235]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:04:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:04:59 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:04:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:59 smithi098 ceph-mon[35848]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:04:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:04:59 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:04:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:59 smithi153 ceph-mon[35521]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:04:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:04:59 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 -- 172.21.15.78:0/3994331557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102350 msgr2=0x7f55f81027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 --2- 172.21.15.78:0/3994331557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102350 0x7f55f81027d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0098f0 tx=0x7f55ec009c00 comp rx=0 tx=0).stop 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 -- 172.21.15.78:0/3994331557 shutdown_connections 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 --2- 172.21.15.78:0/3994331557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55f8102d10 0x7f55f810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 --2- 172.21.15.78:0/3994331557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f81083a0 0x7f55f8108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 --2- 172.21.15.78:0/3994331557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102350 0x7f55f81027d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 -- 172.21.15.78:0/3994331557 >> 172.21.15.78:0/3994331557 conn(0x7f55f8079390 msgr2=0x7f55f80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:04:59.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 -- 172.21.15.78:0/3994331557 shutdown_connections 2024-03-16T11:04:59.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.901+0000 7f55fcde4700 1 -- 172.21.15.78:0/3994331557 wait complete. 2024-03-16T11:04:59.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 Processor -- start 2024-03-16T11:04:59.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 -- start start 2024-03-16T11:04:59.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55f8102350 0x7f55f819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102d10 0x7f55f819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f81083a0 0x7f55f81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8112450 con 0x7f55f8102d10 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81122d0 con 0x7f55f81083a0 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55fcde4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81125d0 con 0x7f55f8102350 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55f5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102d10 0x7f55f819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55f5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102d10 0x7f55f819eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45464/0 (socket says 172.21.15.78:45464) 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55f5d9b700 1 -- 172.21.15.78:0/1747960988 learned_addr learned my addr 172.21.15.78:0/1747960988 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55f6d9d700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f81083a0 0x7f55f81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.902+0000 7f55f659c700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55f8102350 0x7f55f819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55f5d9b700 1 -- 172.21.15.78:0/1747960988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55f8102350 msgr2=0x7f55f819e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55f5d9b700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55f8102350 0x7f55f819e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55f5d9b700 1 -- 172.21.15.78:0/1747960988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f81083a0 msgr2=0x7f55f81987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55f5d9b700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f81083a0 0x7f55f81987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:04:59.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55f5d9b700 1 -- 172.21.15.78:0/1747960988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f8198f90 con 0x7f55f8102d10 2024-03-16T11:04:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55f5d9b700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102d10 0x7f55f819eb70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00f990 tx=0x7f55ec00f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55e77fe700 1 -- 172.21.15.78:0/1747960988 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec027070 con 0x7f55f8102d10 2024-03-16T11:04:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f8199220 con 0x7f55f8102d10 2024-03-16T11:04:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55e77fe700 1 -- 172.21.15.78:0/1747960988 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55ec005570 con 0x7f55f8102d10 2024-03-16T11:04:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55e77fe700 1 -- 172.21.15.78:0/1747960988 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec004930 con 0x7f55f8102d10 2024-03-16T11:04:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.903+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f81a5490 con 0x7f55f8102d10 2024-03-16T11:04:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.904+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f8103a10 con 0x7f55f8102d10 2024-03-16T11:04:59.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.905+0000 7f55e77fe700 1 -- 172.21.15.78:0/1747960988 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f55ec02a070 con 0x7f55f8102d10 2024-03-16T11:04:59.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.906+0000 7f55e77fe700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f55e00a2920 0x7f55e00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:04:59.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.906+0000 7f55f659c700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f55e00a2920 0x7f55e00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:04:59.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.906+0000 7f55e77fe700 1 -- 172.21.15.78:0/1747960988 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f55ec0cf430 con 0x7f55f8102d10 2024-03-16T11:04:59.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.906+0000 7f55f659c700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f55e00a2920 0x7f55e00a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009750 tx=0x7f55e8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:04:59.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:04:59.910+0000 7f55e77fe700 1 -- 172.21.15.78:0/1747960988 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55ec05e740 con 0x7f55f8102d10 2024-03-16T11:05:00.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.108+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f55f802db70 con 0x7f55f8102d10 2024-03-16T11:05:00.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.109+0000 7f55e77fe700 1 -- 172.21.15.78:0/1747960988 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f55ec013030 con 0x7f55f8102d10 2024-03-16T11:05:00.109 INFO:teuthology.orchestra.run.smithi078.stdout:261993005096 2024-03-16T11:05:00.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f55e00a2920 msgr2=0x7f55e00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:00.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f55e00a2920 0x7f55e00a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009750 tx=0x7f55e8009310 comp rx=0 tx=0).stop 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102d10 msgr2=0x7f55f819eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102d10 0x7f55f819eb70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00f990 tx=0x7f55ec00f9c0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 shutdown_connections 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f55e00a2920 0x7f55e00a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55f8102350 0x7f55f819e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f81083a0 0x7f55f81987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 --2- 172.21.15.78:0/1747960988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f8102d10 0x7f55f819eb70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 >> 172.21.15.78:0/1747960988 conn(0x7f55f8079390 msgr2=0x7f55f8101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 shutdown_connections 2024-03-16T11:05:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.111+0000 7f55fcde4700 1 -- 172.21.15.78:0/1747960988 wait complete. 2024-03-16T11:05:00.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 -- 172.21.15.78:0/2104812128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806a9f0 msgr2=0x7f862806ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:00.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 --2- 172.21.15.78:0/2104812128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806a9f0 0x7f862806ae50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f86180099c0 tx=0x7f8618009cd0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 -- 172.21.15.78:0/2104812128 shutdown_connections 2024-03-16T11:05:00.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 --2- 172.21.15.78:0/2104812128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f862806b390 0x7f862810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 --2- 172.21.15.78:0/2104812128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8628108300 0x7f86281086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 --2- 172.21.15.78:0/2104812128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806a9f0 0x7f862806ae50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 -- 172.21.15.78:0/2104812128 >> 172.21.15.78:0/2104812128 conn(0x7f8628078170 msgr2=0x7f8628078580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 -- 172.21.15.78:0/2104812128 shutdown_connections 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.965+0000 7f862f2d6700 1 -- 172.21.15.78:0/2104812128 wait complete. 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 Processor -- start 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 -- start start 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862806a9f0 0x7f8628101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806b390 0x7f86281017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8628108300 0x7f8628101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8628112430 con 0x7f862806b390 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86281122b0 con 0x7f862806a9f0 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862f2d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86281125b0 con 0x7f8628108300 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862c871700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806b390 0x7f86281017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862c871700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806b390 0x7f86281017c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45480/0 (socket says 172.21.15.78:45480) 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.966+0000 7f862c871700 1 -- 172.21.15.78:0/1773381212 learned_addr learned my addr 172.21.15.78:0/1773381212 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862d072700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862806a9f0 0x7f8628101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862d873700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8628108300 0x7f8628101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862c871700 1 -- 172.21.15.78:0/1773381212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8628108300 msgr2=0x7f8628101d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862c871700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8628108300 0x7f8628101d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862c871700 1 -- 172.21.15.78:0/1773381212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862806a9f0 msgr2=0x7f8628101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862c871700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862806a9f0 0x7f8628101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862c871700 1 -- 172.21.15.78:0/1773381212 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86281a6e90 con 0x7f862806b390 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862d873700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8628108300 0x7f8628101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862c871700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806b390 0x7f86281017c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8618006010 tx=0x7f8618009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f861e7fc700 1 -- 172.21.15.78:0/1773381212 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618027070 con 0x7f862806b390 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f861e7fc700 1 -- 172.21.15.78:0/1773381212 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8618003f50 con 0x7f862806b390 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86281a7120 con 0x7f862806b390 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f861e7fc700 1 -- 172.21.15.78:0/1773381212 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618022cd0 con 0x7f862806b390 2024-03-16T11:05:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.967+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86281a7570 con 0x7f862806b390 2024-03-16T11:05:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.968+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86281039b0 con 0x7f862806b390 2024-03-16T11:05:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.969+0000 7f861e7fc700 1 -- 172.21.15.78:0/1773381212 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f861802a070 con 0x7f862806b390 2024-03-16T11:05:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.970+0000 7f861e7fc700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f86140a2920 0x7f86140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.970+0000 7f862d072700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f86140a2920 0x7f86140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.970+0000 7f861e7fc700 1 -- 172.21.15.78:0/1773381212 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f86180cf830 con 0x7f862806b390 2024-03-16T11:05:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.970+0000 7f862d072700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f86140a2920 0x7f86140a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8624005950 tx=0x7f862400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:00.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:00.974+0000 7f861e7fc700 1 -- 172.21.15.78:0/1773381212 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f861805e740 con 0x7f862806b390 2024-03-16T11:05:01.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 -- 172.21.15.78:0/1421903266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 msgr2=0x7f357010e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 --2- 172.21.15.78:0/1421903266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f357010e050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f356800b0a0 tx=0x7f356800b3b0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 -- 172.21.15.78:0/1421903266 shutdown_connections 2024-03-16T11:05:01.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 --2- 172.21.15.78:0/1421903266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3570107210 0x7f35701075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 --2- 172.21.15.78:0/1421903266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f357010e050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 --2- 172.21.15.78:0/1421903266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35701011d0 0x7f3570101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 -- 172.21.15.78:0/1421903266 >> 172.21.15.78:0/1421903266 conn(0x7f3570078100 msgr2=0x7f3570078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:01.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 -- 172.21.15.78:0/1421903266 shutdown_connections 2024-03-16T11:05:01.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 -- 172.21.15.78:0/1421903266 wait complete. 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.169+0000 7f3575617700 1 Processor -- start 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f3575617700 1 -- start start 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f3575617700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35701011d0 0x7f3570199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f3575617700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f3570199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f3575617700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3570107210 0x7f35701a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f3575617700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3570111340 con 0x7f3570101b70 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f3575617700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35701111c0 con 0x7f35701011d0 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f3575617700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35701114c0 con 0x7f3570107210 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f3570199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f3570199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45502/0 (socket says 172.21.15.78:45502) 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 -- 172.21.15.78:0/1889886146 learned_addr learned my addr 172.21.15.78:0/1889886146 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356f7fe700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3570107210 0x7f35701a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356effd700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35701011d0 0x7f3570199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 -- 172.21.15.78:0/1889886146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3570107210 msgr2=0x7f35701a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3570107210 0x7f35701a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 -- 172.21.15.78:0/1889886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35701011d0 msgr2=0x7f3570199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35701011d0 0x7f3570199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.170+0000 7f356e7fc700 1 -- 172.21.15.78:0/1889886146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35701a4d20 con 0x7f3570101b70 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.171+0000 7f356e7fc700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f3570199ca0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f356000e9a0 tx=0x7f356000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.171+0000 7f3557fff700 1 -- 172.21.15.78:0/1889886146 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356000ca10 con 0x7f3570101b70 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.171+0000 7f3557fff700 1 -- 172.21.15.78:0/1889886146 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f356000cbb0 con 0x7f3570101b70 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.171+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35701a4f50 con 0x7f3570101b70 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.171+0000 7f3557fff700 1 -- 172.21.15.78:0/1889886146 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560018680 con 0x7f3570101b70 2024-03-16T11:05:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.171+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35701a5400 con 0x7f3570101b70 2024-03-16T11:05:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.172+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3570068f90 con 0x7f3570101b70 2024-03-16T11:05:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.173+0000 7f3557fff700 1 -- 172.21.15.78:0/1889886146 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3560010430 con 0x7f3570101b70 2024-03-16T11:05:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.174+0000 7f3557fff700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35580a29a0 0x7f35580a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.174+0000 7f356effd700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35580a29a0 0x7f35580a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.174+0000 7f3557fff700 1 -- 172.21.15.78:0/1889886146 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f35600cee90 con 0x7f3570101b70 2024-03-16T11:05:01.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.175+0000 7f356effd700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35580a29a0 0x7f35580a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3564009be0 tx=0x7f3564009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:01.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.178+0000 7f3557fff700 1 -- 172.21.15.78:0/1889886146 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f356009be80 con 0x7f3570101b70 2024-03-16T11:05:01.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.182+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f862802db70 con 0x7f862806b390 2024-03-16T11:05:01.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.182+0000 7f861e7fc700 1 -- 172.21.15.78:0/1773381212 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8618096e20 con 0x7f862806b390 2024-03-16T11:05:01.183 INFO:teuthology.orchestra.run.smithi078.stdout:146028888123 2024-03-16T11:05:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f86140a2920 msgr2=0x7f86140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f86140a2920 0x7f86140a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8624005950 tx=0x7f862400b410 comp rx=0 tx=0).stop 2024-03-16T11:05:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806b390 msgr2=0x7f86281017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806b390 0x7f86281017c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8618006010 tx=0x7f8618009f80 comp rx=0 tx=0).stop 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 shutdown_connections 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f86140a2920 0x7f86140a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8628108300 0x7f8628101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862806a9f0 0x7f8628101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 --2- 172.21.15.78:0/1773381212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f862806b390 0x7f86281017c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 >> 172.21.15.78:0/1773381212 conn(0x7f8628078170 msgr2=0x7f86280ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 shutdown_connections 2024-03-16T11:05:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.185+0000 7f862f2d6700 1 -- 172.21.15.78:0/1773381212 wait complete. 2024-03-16T11:05:01.375 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:01 smithi078 ceph-mon[32235]: pgmap v317: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:01.375 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:01 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1747960988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T11:05:01.375 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.375 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.375 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.375 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:01 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.391+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f357002dd30 con 0x7f3570101b70 2024-03-16T11:05:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f3557fff700 1 -- 172.21.15.78:0/1889886146 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f356009b780 con 0x7f3570101b70 2024-03-16T11:05:01.392 INFO:teuthology.orchestra.run.smithi078.stdout:34359738448 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 -- 172.21.15.78:0/2817702659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18341075f0 msgr2=0x7f183410f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 --2- 172.21.15.78:0/2817702659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18341075f0 0x7f183410f1c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f182c00b0d0 tx=0x7f182c00b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 -- 172.21.15.78:0/2817702659 shutdown_connections 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 --2- 172.21.15.78:0/2817702659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 0x7f18341028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 --2- 172.21.15.78:0/2817702659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18341075f0 0x7f183410f1c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 --2- 172.21.15.78:0/2817702659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834102ec0 0x7f1834106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 -- 172.21.15.78:0/2817702659 >> 172.21.15.78:0/2817702659 conn(0x7f18340fdd80 msgr2=0x7f18341001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 -- 172.21.15.78:0/2817702659 shutdown_connections 2024-03-16T11:05:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.392+0000 7f183a4c5700 1 -- 172.21.15.78:0/2817702659 wait complete. 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 Processor -- start 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 -- start start 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 0x7f18340745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1834102ec0 0x7f1834074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341075f0 0x7f1834075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f18341124b0 con 0x7f1834102ec0 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1834112330 con 0x7f18341075f0 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.393+0000 7f183a4c5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1834112630 con 0x7f1834102510 2024-03-16T11:05:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f1833fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 0x7f18340745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1834102ec0 0x7f1834074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f1833fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 0x7f18340745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:47584/0 (socket says 172.21.15.78:47584) 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f1833fff700 1 -- 172.21.15.78:0/508268318 learned_addr learned my addr 172.21.15.78:0/508268318 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f1838a62700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341075f0 0x7f1834075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18337fe700 1 -- 172.21.15.78:0/508268318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 msgr2=0x7f18340745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18337fe700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 0x7f18340745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18337fe700 1 -- 172.21.15.78:0/508268318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341075f0 msgr2=0x7f1834075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18337fe700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341075f0 0x7f1834075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18337fe700 1 -- 172.21.15.78:0/508268318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18341a6e80 con 0x7f1834102ec0 2024-03-16T11:05:01.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f1833fff700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 0x7f18340745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:01.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f1838a62700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341075f0 0x7f1834075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:01.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18337fe700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1834102ec0 0x7f1834074b10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1834069d20 tx=0x7f182400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:01.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18317fa700 1 -- 172.21.15.78:0/508268318 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824016070 con 0x7f1834102ec0 2024-03-16T11:05:01.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18317fa700 1 -- 172.21.15.78:0/508268318 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f182400fd40 con 0x7f1834102ec0 2024-03-16T11:05:01.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18341a7170 con 0x7f1834102ec0 2024-03-16T11:05:01.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f18317fa700 1 -- 172.21.15.78:0/508268318 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182400e3b0 con 0x7f1834102ec0 2024-03-16T11:05:01.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.394+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18341a7700 con 0x7f1834102ec0 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35580a29a0 msgr2=0x7f35580a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35580a29a0 0x7f35580a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3564009be0 tx=0x7f3564009260 comp rx=0 tx=0).stop 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 msgr2=0x7f3570199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f3570199ca0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f356000e9a0 tx=0x7f356000ecb0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 shutdown_connections 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35580a29a0 0x7f35580a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3570107210 0x7f35701a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35701011d0 0x7f3570199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 --2- 172.21.15.78:0/1889886146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3570101b70 0x7f3570199ca0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 >> 172.21.15.78:0/1889886146 conn(0x7f3570078100 msgr2=0x7f35700fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:01.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 shutdown_connections 2024-03-16T11:05:01.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f3575617700 1 -- 172.21.15.78:0/1889886146 wait complete. 2024-03-16T11:05:01.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.395+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1834068f90 con 0x7f1834102ec0 2024-03-16T11:05:01.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.397+0000 7f18317fa700 1 -- 172.21.15.78:0/508268318 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1824019070 con 0x7f1834102ec0 2024-03-16T11:05:01.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.397+0000 7f18317fa700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f181c0a29a0 0x7f181c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:01.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.397+0000 7f1833fff700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f181c0a29a0 0x7f181c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:01.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.397+0000 7f18317fa700 1 -- 172.21.15.78:0/508268318 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f18240ce7c0 con 0x7f1834102ec0 2024-03-16T11:05:01.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.398+0000 7f1833fff700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f181c0a29a0 0x7f181c0a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1828011590 tx=0x7f1828009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:01.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.401+0000 7f18317fa700 1 -- 172.21.15.78:0/508268318 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1824010460 con 0x7f1834102ec0 2024-03-16T11:05:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:01 smithi098 ceph-mon[35848]: pgmap v317: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:01 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1747960988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T11:05:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:01 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:01 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:01 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:01 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:01 smithi153 ceph-mon[35521]: pgmap v317: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:01 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1747960988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T11:05:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:01 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.609+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f1834075ab0 con 0x7f1834102ec0 2024-03-16T11:05:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.609+0000 7f18317fa700 1 -- 172.21.15.78:0/508268318 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f182409b8b0 con 0x7f1834102ec0 2024-03-16T11:05:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:356482285593 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f181c0a29a0 msgr2=0x7f181c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f181c0a29a0 0x7f181c0a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1828011590 tx=0x7f1828009260 comp rx=0 tx=0).stop 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1834102ec0 msgr2=0x7f1834074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1834102ec0 0x7f1834074b10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1834069d20 tx=0x7f182400dc30 comp rx=0 tx=0).stop 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 shutdown_connections 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f181c0a29a0 0x7f181c0a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834102510 0x7f18340745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341075f0 0x7f1834075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 --2- 172.21.15.78:0/508268318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1834102ec0 0x7f1834074b10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 >> 172.21.15.78:0/508268318 conn(0x7f18340fdd80 msgr2=0x7f1834100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 shutdown_connections 2024-03-16T11:05:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:01.611+0000 7f183a4c5700 1 -- 172.21.15.78:0/508268318 wait complete. 2024-03-16T11:05:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1773381212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T11:05:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1889886146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T11:05:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/508268318' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T11:05:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-16T11:05:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-16T11:05:02.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T11:05:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-16T11:05:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-03-16T11:05:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-16T11:05:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-16T11:05:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-16T11:05:02.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:02 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-16T11:05:02.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1773381212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T11:05:02.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1889886146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T11:05:02.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/508268318' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T11:05:02.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-16T11:05:02.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:02 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-16T11:05:02.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 -- 172.21.15.78:0/3489147441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 msgr2=0x7f743c10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:02.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 --2- 172.21.15.78:0/3489147441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c10f160 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f743400b0d0 tx=0x7f743400b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 -- 172.21.15.78:0/3489147441 shutdown_connections 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 --2- 172.21.15.78:0/3489147441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c10f160 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 --2- 172.21.15.78:0/3489147441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c108320 0x7f743c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 --2- 172.21.15.78:0/3489147441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f743c069f80 0x7f743c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 -- 172.21.15.78:0/3489147441 >> 172.21.15.78:0/3489147441 conn(0x7f743c078100 msgr2=0x7f743c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 -- 172.21.15.78:0/3489147441 shutdown_connections 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 -- 172.21.15.78:0/3489147441 wait complete. 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.520+0000 7f74410d6700 1 Processor -- start 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f74410d6700 1 -- start start 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f74410d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f743c069f80 0x7f743c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f74410d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f74410d6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c108320 0x7f743c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f74410d6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f743c112450 con 0x7f743c06a920 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f74410d6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f743c1122d0 con 0x7f743c108320 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f74410d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f743c1125d0 con 0x7f743c069f80 2024-03-16T11:05:02.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45542/0 (socket says 172.21.15.78:45542) 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 -- 172.21.15.78:0/2835450223 learned_addr learned my addr 172.21.15.78:0/2835450223 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743ad9d700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f743c069f80 0x7f743c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743b59e700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c108320 0x7f743c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 -- 172.21.15.78:0/2835450223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f743c069f80 msgr2=0x7f743c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f743c069f80 0x7f743c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 -- 172.21.15.78:0/2835450223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c108320 msgr2=0x7f743c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c108320 0x7f743c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 -- 172.21.15.78:0/2835450223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743c198ff0 con 0x7f743c06a920 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.521+0000 7f743259c700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c19eb80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f742400e9c0 tx=0x7f742400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.522+0000 7f7438d99700 1 -- 172.21.15.78:0/2835450223 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742400cbd0 con 0x7f743c06a920 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.522+0000 7f7438d99700 1 -- 172.21.15.78:0/2835450223 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f742400ee60 con 0x7f743c06a920 2024-03-16T11:05:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.522+0000 7f7438d99700 1 -- 172.21.15.78:0/2835450223 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74240055b0 con 0x7f743c06a920 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.522+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743c1992e0 con 0x7f743c06a920 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.522+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f743c1a54a0 con 0x7f743c06a920 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.523+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f743c1039d0 con 0x7f743c06a920 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.524+0000 7f7438d99700 1 -- 172.21.15.78:0/2835450223 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7424004750 con 0x7f743c06a920 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.524+0000 7f7438d99700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74280a2920 0x7f74280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.524+0000 7f743ad9d700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74280a2920 0x7f74280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.524+0000 7f7438d99700 1 -- 172.21.15.78:0/2835450223 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f74240cee60 con 0x7f743c06a920 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.525+0000 7f743ad9d700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74280a2920 0x7f74280a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f742c009be0 tx=0x7f742c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.529+0000 7f7438d99700 1 -- 172.21.15.78:0/2835450223 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f742409bda0 con 0x7f743c06a920 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1773381212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1889886146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/508268318' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-16T11:05:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-03-16T11:05:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-16T11:05:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-16T11:05:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-16T11:05:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:02 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-16T11:05:02.677 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005090 got 261993005096 for osd.8 2024-03-16T11:05:02.677 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:02.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.738+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f743c02db70 con 0x7f743c06a920 2024-03-16T11:05:02.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.738+0000 7f7438d99700 1 -- 172.21.15.78:0/2835450223 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f742409b6a0 con 0x7f743c06a920 2024-03-16T11:05:02.739 INFO:teuthology.orchestra.run.smithi078.stdout:227633266733 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74280a2920 msgr2=0x7f74280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74280a2920 0x7f74280a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f742c009be0 tx=0x7f742c009260 comp rx=0 tx=0).stop 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 msgr2=0x7f743c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c19eb80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f742400e9c0 tx=0x7f742400ecd0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 shutdown_connections 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74280a2920 0x7f74280a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f743c069f80 0x7f743c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c108320 0x7f743c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 --2- 172.21.15.78:0/2835450223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f743c06a920 0x7f743c19eb80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 >> 172.21.15.78:0/2835450223 conn(0x7f743c078100 msgr2=0x7f743c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 shutdown_connections 2024-03-16T11:05:02.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:02.740+0000 7f74410d6700 1 -- 172.21.15.78:0/2835450223 wait complete. 2024-03-16T11:05:03.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: pgmap v318: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T11:05:03.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:03 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2835450223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T11:05:03.406 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285588 got 356482285593 for osd.11 2024-03-16T11:05:03.406 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: pgmap v318: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-16T11:05:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-16T11:05:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T11:05:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-16T11:05:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-03-16T11:05:03.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-16T11:05:03.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-16T11:05:03.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-16T11:05:03.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-16T11:05:03.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T11:05:03.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:03 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2835450223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T11:05:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: pgmap v318: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-16T11:05:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-16T11:05:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T11:05:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-16T11:05:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-03-16T11:05:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-16T11:05:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-16T11:05:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-16T11:05:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-16T11:05:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T11:05:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:03 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2835450223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T11:05:03.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 -- 172.21.15.78:0/533550268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 msgr2=0x7fa270107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:03.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 --2- 172.21.15.78:0/533550268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270107640 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa260009990 tx=0x7fa260009ca0 comp rx=0 tx=0).stop 2024-03-16T11:05:03.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 -- 172.21.15.78:0/533550268 shutdown_connections 2024-03-16T11:05:03.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 --2- 172.21.15.78:0/533550268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa270101b70 0x7fa27010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:03.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 --2- 172.21.15.78:0/533550268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270107640 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:03.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 --2- 172.21.15.78:0/533550268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2701011b0 0x7fa270101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:03.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 -- 172.21.15.78:0/533550268 >> 172.21.15.78:0/533550268 conn(0x7fa2700781e0 msgr2=0x7fa2700785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:03.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 -- 172.21.15.78:0/533550268 shutdown_connections 2024-03-16T11:05:03.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.825+0000 7fa275a00700 1 -- 172.21.15.78:0/533550268 wait complete. 2024-03-16T11:05:03.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 Processor -- start 2024-03-16T11:05:03.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 -- start start 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2701011b0 0x7fa27019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa270101b70 0x7fa27019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa270111350 con 0x7fa270107260 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2701111d0 con 0x7fa270101b70 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.826+0000 7fa275a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa2701114d0 con 0x7fa2701011b0 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa270101b70 0x7fa27019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45574/0 (socket says 172.21.15.78:45574) 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26f7fe700 1 -- 172.21.15.78:0/3491280984 learned_addr learned my addr 172.21.15.78:0/3491280984 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26effd700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2701011b0 0x7fa27019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26e7fc700 1 -- 172.21.15.78:0/3491280984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2701011b0 msgr2=0x7fa27019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:03.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26e7fc700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2701011b0 0x7fa27019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26e7fc700 1 -- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 msgr2=0x7fa270197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26e7fc700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26e7fc700 1 -- 172.21.15.78:0/3491280984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa270197d70 con 0x7fa270101b70 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26f7fe700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa26e7fc700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa270101b70 0x7fa27019d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa26400e9f0 tx=0x7fa26400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa2749fe700 1 -- 172.21.15.78:0/3491280984 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26400c9c0 con 0x7fa270101b70 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa270198060 con 0x7fa270101b70 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa2749fe700 1 -- 172.21.15.78:0/3491280984 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa26400cb60 con 0x7fa270101b70 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa2749fe700 1 -- 172.21.15.78:0/3491280984 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264018680 con 0x7fa270101b70 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.827+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2701a4320 con 0x7fa270101b70 2024-03-16T11:05:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.828+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa270068f90 con 0x7fa270101b70 2024-03-16T11:05:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.829+0000 7fa2749fe700 1 -- 172.21.15.78:0/3491280984 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa264047a90 con 0x7fa270101b70 2024-03-16T11:05:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.830+0000 7fa2749fe700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa2580a2920 0x7fa2580a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.830+0000 7fa26effd700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa2580a2920 0x7fa2580a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.830+0000 7fa2749fe700 1 -- 172.21.15.78:0/3491280984 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fa2640cee00 con 0x7fa270101b70 2024-03-16T11:05:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.830+0000 7fa26effd700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa2580a2920 0x7fa2580a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa26000b2e0 tx=0x7fa260005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:03.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:03.833+0000 7fa2749fe700 1 -- 172.21.15.78:0/3491280984 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa26409bd00 con 0x7fa270101b70 2024-03-16T11:05:04.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.032+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa27002db70 con 0x7fa270101b70 2024-03-16T11:05:04.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.033+0000 7fa2749fe700 1 -- 172.21.15.78:0/3491280984 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa26409b600 con 0x7fa270101b70 2024-03-16T11:05:04.033 INFO:teuthology.orchestra.run.smithi078.stdout:55834574923 2024-03-16T11:05:04.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa2580a2920 msgr2=0x7fa2580a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:04.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa2580a2920 0x7fa2580a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa26000b2e0 tx=0x7fa260005960 comp rx=0 tx=0).stop 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa270101b70 msgr2=0x7fa27019d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa270101b70 0x7fa27019d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa26400e9f0 tx=0x7fa26400ed00 comp rx=0 tx=0).stop 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 shutdown_connections 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa2580a2920 0x7fa2580a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2701011b0 0x7fa27019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa270107260 0x7fa270197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 --2- 172.21.15.78:0/3491280984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa270101b70 0x7fa27019d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 >> 172.21.15.78:0/3491280984 conn(0x7fa2700781e0 msgr2=0x7fa2701000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 shutdown_connections 2024-03-16T11:05:04.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:04.035+0000 7fa275a00700 1 -- 172.21.15.78:0/3491280984 wait complete. 2024-03-16T11:05:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:04 smithi078 ceph-mon[32235]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T11:05:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:04 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3491280984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T11:05:04.296 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266729 got 227633266733 for osd.7 2024-03-16T11:05:04.296 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:04.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:04 smithi098 ceph-mon[35848]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T11:05:04.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:04 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3491280984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T11:05:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:04.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:04 smithi153 ceph-mon[35521]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T11:05:04.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:04 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3491280984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T11:05:04.867 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738443 got 34359738448 for osd.0 2024-03-16T11:05:04.867 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:05.331 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888119 got 146028888123 for osd.4 2024-03-16T11:05:05.332 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:05.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:05 smithi078 ceph-mon[32235]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 14 B/s, 0 objects/s recovering 2024-03-16T11:05:05.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:05 smithi098 ceph-mon[35848]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 14 B/s, 0 objects/s recovering 2024-03-16T11:05:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:05 smithi153 ceph-mon[35521]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 14 B/s, 0 objects/s recovering 2024-03-16T11:05:06.312 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 -- 172.21.15.78:0/2958516676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806b420 msgr2=0x7f9088106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 --2- 172.21.15.78:0/2958516676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806b420 0x7f9088106f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f907c0099c0 tx=0x7f907c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 -- 172.21.15.78:0/2958516676 shutdown_connections 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 --2- 172.21.15.78:0/2958516676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9088107650 0x7f908810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 --2- 172.21.15.78:0/2958516676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f908806aa70 0x7f908806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 --2- 172.21.15.78:0/2958516676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806b420 0x7f9088106f20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 -- 172.21.15.78:0/2958516676 >> 172.21.15.78:0/2958516676 conn(0x7f9088077220 msgr2=0x7f9088077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 -- 172.21.15.78:0/2958516676 shutdown_connections 2024-03-16T11:05:06.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.334+0000 7f908e182700 1 -- 172.21.15.78:0/2958516676 wait complete. 2024-03-16T11:05:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.335+0000 7f908e182700 1 Processor -- start 2024-03-16T11:05:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.335+0000 7f908e182700 1 -- start start 2024-03-16T11:05:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f908e182700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806aa70 0x7f90881012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f908e182700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f908806b420 0x7f9088101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f908e182700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9088107650 0x7f9088101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f908e182700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9088112470 con 0x7f908806aa70 2024-03-16T11:05:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f908e182700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90881122f0 con 0x7f9088107650 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f908e182700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90881125f0 con 0x7f908806b420 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f90877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806aa70 0x7f90881012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9088107650 0x7f9088101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9086ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f908806b420 0x7f9088101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9088107650 0x7f9088101d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:53460/0 (socket says 172.21.15.78:53460) 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 -- 172.21.15.78:0/356347949 learned_addr learned my addr 172.21.15.78:0/356347949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 -- 172.21.15.78:0/356347949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f908806b420 msgr2=0x7f9088101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f908806b420 0x7f9088101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 -- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806aa70 msgr2=0x7f90881012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806aa70 0x7f90881012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f9087fff700 1 -- 172.21.15.78:0/356347949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90881a6ed0 con 0x7f9088107650 2024-03-16T11:05:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.336+0000 7f90877fe700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806aa70 0x7f90881012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.337+0000 7f9087fff700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9088107650 0x7f9088101d50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f908000ef30 tx=0x7f908000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.337+0000 7f9084ff9700 1 -- 172.21.15.78:0/356347949 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90800092e0 con 0x7f9088107650 2024-03-16T11:05:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.337+0000 7f9084ff9700 1 -- 172.21.15.78:0/356347949 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9080004bb0 con 0x7f9088107650 2024-03-16T11:05:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.337+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90881a7160 con 0x7f9088107650 2024-03-16T11:05:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.337+0000 7f9084ff9700 1 -- 172.21.15.78:0/356347949 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9080004060 con 0x7f9088107650 2024-03-16T11:05:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.337+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90881a75b0 con 0x7f9088107650 2024-03-16T11:05:06.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.339+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9088068f90 con 0x7f9088107650 2024-03-16T11:05:06.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.339+0000 7f9084ff9700 1 -- 172.21.15.78:0/356347949 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f908001b030 con 0x7f9088107650 2024-03-16T11:05:06.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.339+0000 7f9084ff9700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f90700a2920 0x7f90700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:06.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.340+0000 7f9084ff9700 1 -- 172.21.15.78:0/356347949 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f90800ce730 con 0x7f9088107650 2024-03-16T11:05:06.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.340+0000 7f90877fe700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f90700a2920 0x7f90700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:06.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.340+0000 7f90877fe700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f90700a2920 0x7f90700a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9078011490 tx=0x7f9078009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.344+0000 7f9084ff9700 1 -- 172.21.15.78:0/356347949 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9080022020 con 0x7f9088107650 2024-03-16T11:05:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.550+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f908802db70 con 0x7f9088107650 2024-03-16T11:05:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.550+0000 7f9084ff9700 1 -- 172.21.15.78:0/356347949 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9080010040 con 0x7f9088107650 2024-03-16T11:05:06.551 INFO:teuthology.orchestra.run.smithi078.stdout:77309411399 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.552+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f90700a2920 msgr2=0x7f90700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.552+0000 7f908e182700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f90700a2920 0x7f90700a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9078011490 tx=0x7f9078009260 comp rx=0 tx=0).stop 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.552+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9088107650 msgr2=0x7f9088101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.552+0000 7f908e182700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9088107650 0x7f9088101d50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f908000ef30 tx=0x7f908000c3a0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 shutdown_connections 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f90700a2920 0x7f90700a4de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9088107650 0x7f9088101d50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f908806aa70 0x7f90881012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 --2- 172.21.15.78:0/356347949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f908806b420 0x7f9088101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 >> 172.21.15.78:0/356347949 conn(0x7f9088077220 msgr2=0x7f90880ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 shutdown_connections 2024-03-16T11:05:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:06.553+0000 7f908e182700 1 -- 172.21.15.78:0/356347949 wait complete. 2024-03-16T11:05:06.804 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574923 for osd.1 2024-03-16T11:05:06.805 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:07.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:07 smithi078 ceph-mon[32235]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:07.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:07 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/356347949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T11:05:07.410 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411399 for osd.2 2024-03-16T11:05:07.410 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:07 smithi098 ceph-mon[35848]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:07 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/356347949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T11:05:07.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:07 smithi153 ceph-mon[35521]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:07.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:07 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/356347949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T11:05:09.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:09 smithi078 ceph-mon[32235]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:09 smithi098 ceph-mon[35848]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:09 smithi153 ceph-mon[35521]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:09.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 -- 172.21.15.78:0/111133322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 msgr2=0x7f639410f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:09.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 --2- 172.21.15.78:0/111133322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f639410f210 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f638c00b0d0 tx=0x7f638c00b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 -- 172.21.15.78:0/111133322 shutdown_connections 2024-03-16T11:05:09.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 --2- 172.21.15.78:0/111133322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6394100e70 0x7f6394101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 --2- 172.21.15.78:0/111133322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f639410f210 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 --2- 172.21.15.78:0/111133322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394101820 0x7f6394106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 -- 172.21.15.78:0/111133322 >> 172.21.15.78:0/111133322 conn(0x7f6394076510 msgr2=0x7f6394076920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 -- 172.21.15.78:0/111133322 shutdown_connections 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 -- 172.21.15.78:0/111133322 wait complete. 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 Processor -- start 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 -- start start 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.756+0000 7f639add5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394100e70 0x7f6394101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f639add5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6394101820 0x7f6394102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:09.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f639add5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f6394102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f639add5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6394112500 con 0x7f6394107690 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f639add5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6394112380 con 0x7f6394100e70 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f639add5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6394112680 con 0x7f6394101820 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f6394102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f6394102870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34904/0 (socket says 172.21.15.78:34904) 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 -- 172.21.15.78:0/1618225975 learned_addr learned my addr 172.21.15.78:0/1618225975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6398b71700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394100e70 0x7f6394101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6393fff700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6394101820 0x7f6394102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 -- 172.21.15.78:0/1618225975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6394101820 msgr2=0x7f6394102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6394101820 0x7f6394102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 -- 172.21.15.78:0/1618225975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394100e70 msgr2=0x7f6394101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394100e70 0x7f6394101df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 -- 172.21.15.78:0/1618225975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63941a6e90 con 0x7f6394107690 2024-03-16T11:05:09.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6399372700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f6394102870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f638c007a00 tx=0x7f638c007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:09.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6391ffb700 1 -- 172.21.15.78:0/1618225975 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c01f070 con 0x7f6394107690 2024-03-16T11:05:09.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63941a7120 con 0x7f6394107690 2024-03-16T11:05:09.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6391ffb700 1 -- 172.21.15.78:0/1618225975 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f638c004c50 con 0x7f6394107690 2024-03-16T11:05:09.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.757+0000 7f6391ffb700 1 -- 172.21.15.78:0/1618225975 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c01b690 con 0x7f6394107690 2024-03-16T11:05:09.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.758+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63941a75d0 con 0x7f6394107690 2024-03-16T11:05:09.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.758+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f639410a8d0 con 0x7f6394107690 2024-03-16T11:05:09.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.759+0000 7f6391ffb700 1 -- 172.21.15.78:0/1618225975 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f638c022070 con 0x7f6394107690 2024-03-16T11:05:09.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.760+0000 7f6391ffb700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63840a2920 0x7f63840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:09.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.760+0000 7f6398b71700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63840a2920 0x7f63840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:09.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.760+0000 7f6391ffb700 1 -- 172.21.15.78:0/1618225975 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f638c00f020 con 0x7f6394107690 2024-03-16T11:05:09.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.760+0000 7f6398b71700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63840a2920 0x7f63840a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6388009c60 tx=0x7f6388009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:09.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.764+0000 7f6391ffb700 1 -- 172.21.15.78:0/1618225975 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f638c029020 con 0x7f6394107690 2024-03-16T11:05:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.963+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f639410a2b0 con 0x7f6394107690 2024-03-16T11:05:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.964+0000 7f6391ffb700 1 -- 172.21.15.78:0/1618225975 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f638c09d0e0 con 0x7f6394107690 2024-03-16T11:05:09.964 INFO:teuthology.orchestra.run.smithi078.stdout:171798691898 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63840a2920 msgr2=0x7f63840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63840a2920 0x7f63840a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6388009c60 tx=0x7f6388009260 comp rx=0 tx=0).stop 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 msgr2=0x7f6394102870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f6394102870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f638c007a00 tx=0x7f638c007a30 comp rx=0 tx=0).stop 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 shutdown_connections 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63840a2920 0x7f63840a4de0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6394107690 0x7f6394102870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394100e70 0x7f6394101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 --2- 172.21.15.78:0/1618225975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6394101820 0x7f6394102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 >> 172.21.15.78:0/1618225975 conn(0x7f6394076510 msgr2=0x7f63940fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 shutdown_connections 2024-03-16T11:05:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:09.966+0000 7f639add5700 1 -- 172.21.15.78:0/1618225975 wait complete. 2024-03-16T11:05:10.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:10 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1618225975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T11:05:10.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:10 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1618225975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T11:05:10.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:10 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1618225975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T11:05:10.864 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691892 got 171798691898 for osd.5 2024-03-16T11:05:10.864 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:11.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:11 smithi078 ceph-mon[32235]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 -- 172.21.15.78:0/1011709050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 msgr2=0x7f8b9c1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 --2- 172.21.15.78:0/1011709050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c1028f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8b900099c0 tx=0x7f8b90009cd0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 -- 172.21.15.78:0/1011709050 shutdown_connections 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 --2- 172.21.15.78:0/1011709050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b9c1075f0 0x7f8b9c10f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 --2- 172.21.15.78:0/1011709050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c1028f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 --2- 172.21.15.78:0/1011709050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b9c102ec0 0x7f8b9c106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 -- 172.21.15.78:0/1011709050 >> 172.21.15.78:0/1011709050 conn(0x7f8b9c0fdd80 msgr2=0x7f8b9c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:11.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 -- 172.21.15.78:0/1011709050 shutdown_connections 2024-03-16T11:05:11.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.340+0000 7f8ba3264700 1 -- 172.21.15.78:0/1011709050 wait complete. 2024-03-16T11:05:11.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 Processor -- start 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 -- start start 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b9c102ec0 0x7f8b9c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b9c1075f0 0x7f8b9c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c112480 con 0x7f8b9c102510 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c112300 con 0x7f8b9c102ec0 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba3264700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c112600 con 0x7f8b9c1075f0 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34928/0 (socket says 172.21.15.78:34928) 2024-03-16T11:05:11.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 -- 172.21.15.78:0/2918003257 learned_addr learned my addr 172.21.15.78:0/2918003257 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1801700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b9c1075f0 0x7f8b9c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8b9bfff700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b9c102ec0 0x7f8b9c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 -- 172.21.15.78:0/2918003257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b9c1075f0 msgr2=0x7f8b9c1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b9c1075f0 0x7f8b9c1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 -- 172.21.15.78:0/2918003257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b9c102ec0 msgr2=0x7f8b9c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b9c102ec0 0x7f8b9c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.341+0000 7f8ba1000700 1 -- 172.21.15.78:0/2918003257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c1990a0 con 0x7f8b9c102510 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.342+0000 7f8ba1000700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c19e6a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b90006010 tx=0x7f8b900055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.342+0000 7f8b99ffb700 1 -- 172.21.15.78:0/2918003257 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b90026070 con 0x7f8b9c102510 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.342+0000 7f8b99ffb700 1 -- 172.21.15.78:0/2918003257 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b9000f460 con 0x7f8b9c102510 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.342+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c199330 con 0x7f8b9c102510 2024-03-16T11:05:11.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.342+0000 7f8b99ffb700 1 -- 172.21.15.78:0/2918003257 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b90009e60 con 0x7f8b9c102510 2024-03-16T11:05:11.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.342+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9c1a5450 con 0x7f8b9c102510 2024-03-16T11:05:11.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.343+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b9c1039f0 con 0x7f8b9c102510 2024-03-16T11:05:11.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.344+0000 7f8b99ffb700 1 -- 172.21.15.78:0/2918003257 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8b90029070 con 0x7f8b9c102510 2024-03-16T11:05:11.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.345+0000 7f8b99ffb700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2920 0x7f8b8c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:11.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.345+0000 7f8b9bfff700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2920 0x7f8b8c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:11.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.345+0000 7f8b99ffb700 1 -- 172.21.15.78:0/2918003257 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8b900cf7f0 con 0x7f8b9c102510 2024-03-16T11:05:11.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.345+0000 7f8b9bfff700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2920 0x7f8b8c0a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c199f30 tx=0x7f8b88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:11.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.348+0000 7f8b99ffb700 1 -- 172.21.15.78:0/2918003257 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b90030020 con 0x7f8b9c102510 2024-03-16T11:05:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:11 smithi098 ceph-mon[35848]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:11 smithi153 ceph-mon[35521]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 17 B/s, 0 objects/s recovering 2024-03-16T11:05:11.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.596+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f8b9c02db70 con 0x7f8b9c102510 2024-03-16T11:05:11.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.596+0000 7f8b99ffb700 1 -- 172.21.15.78:0/2918003257 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8b90013070 con 0x7f8b9c102510 2024-03-16T11:05:11.597 INFO:teuthology.orchestra.run.smithi078.stdout:296352743462 2024-03-16T11:05:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.598+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2920 msgr2=0x7f8b8c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.598+0000 7f8ba3264700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2920 0x7f8b8c0a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c199f30 tx=0x7f8b88009260 comp rx=0 tx=0).stop 2024-03-16T11:05:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.598+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 msgr2=0x7f8b9c19e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.598+0000 7f8ba3264700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c19e6a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b90006010 tx=0x7f8b900055f0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 shutdown_connections 2024-03-16T11:05:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b8c0a2920 0x7f8b8c0a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b9c1075f0 0x7f8b9c1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b9c102510 0x7f8b9c19e6a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 --2- 172.21.15.78:0/2918003257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b9c102ec0 0x7f8b9c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 >> 172.21.15.78:0/2918003257 conn(0x7f8b9c0fdd80 msgr2=0x7f8b9c1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 shutdown_connections 2024-03-16T11:05:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:11.599+0000 7f8ba3264700 1 -- 172.21.15.78:0/2918003257 wait complete. 2024-03-16T11:05:12.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:12 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2918003257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T11:05:12.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:12.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:12 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:12.227 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743458 got 296352743462 for osd.9 2024-03-16T11:05:12.227 DEBUG:teuthology.parallel:result is None 2024-03-16T11:05:12.227 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-16T11:05:12.227 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T11:05:12.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:12 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2918003257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T11:05:12.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:12.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:12 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:12.490 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:12.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:12 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2918003257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T11:05:12.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:12.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:12 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:13 smithi153 ceph-mon[35521]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2024-03-16T11:05:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:05:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:05:13.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:13 smithi098 ceph-mon[35848]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2024-03-16T11:05:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:05:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:05:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:13.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:13 smithi078 ceph-mon[32235]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2024-03-16T11:05:13.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:05:13.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:05:13.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 -- 172.21.15.78:0/3129812629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70102ce0 msgr2=0x7faa7010f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 --2- 172.21.15.78:0/3129812629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70102ce0 0x7faa7010f120 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faa6800b0d0 tx=0x7faa6800b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 -- 172.21.15.78:0/3129812629 shutdown_connections 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 --2- 172.21.15.78:0/3129812629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70102ce0 0x7faa7010f120 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 --2- 172.21.15.78:0/3129812629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70108300 0x7faa701086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 --2- 172.21.15.78:0/3129812629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 0x7faa701027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 -- 172.21.15.78:0/3129812629 >> 172.21.15.78:0/3129812629 conn(0x7faa70079400 msgr2=0x7faa70079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 -- 172.21.15.78:0/3129812629 shutdown_connections 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 -- 172.21.15.78:0/3129812629 wait complete. 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 Processor -- start 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.134+0000 7faa77532700 1 -- start start 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa77532700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 0x7faa7019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa77532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70102ce0 0x7faa7019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa77532700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70108300 0x7faa701986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa77532700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7faa70112460 con 0x7faa70108300 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa77532700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa701122e0 con 0x7faa70102ce0 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa77532700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faa701125e0 con 0x7faa70102320 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 0x7faa7019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa74acd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70102ce0 0x7faa7019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 0x7faa7019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:56668/0 (socket says 172.21.15.78:56668) 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 -- 172.21.15.78:0/3985609063 learned_addr learned my addr 172.21.15.78:0/3985609063 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa75acf700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70108300 0x7faa701986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 -- 172.21.15.78:0/3985609063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70102ce0 msgr2=0x7faa7019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70102ce0 0x7faa7019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 -- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70108300 msgr2=0x7faa701986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70108300 0x7faa701986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 -- 172.21.15.78:0/3985609063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa70198f80 con 0x7faa70102320 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa74acd700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70102ce0 0x7faa7019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa752ce700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 0x7faa7019e5d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00eb30 tx=0x7faa6c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.135+0000 7faa667fc700 1 -- 172.21.15.78:0/3985609063 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c00cbd0 con 0x7faa70102320 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.136+0000 7faa667fc700 1 -- 172.21.15.78:0/3985609063 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa6c0044d0 con 0x7faa70102320 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.136+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa70199270 con 0x7faa70102320 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.136+0000 7faa667fc700 1 -- 172.21.15.78:0/3985609063 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c003e30 con 0x7faa70102320 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.136+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa701a5550 con 0x7faa70102320 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.136+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa701039b0 con 0x7faa70102320 2024-03-16T11:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.138+0000 7faa667fc700 1 -- 172.21.15.78:0/3985609063 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7faa6c004000 con 0x7faa70102320 2024-03-16T11:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.138+0000 7faa667fc700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa5c0a2920 0x7faa5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.138+0000 7faa74acd700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa5c0a2920 0x7faa5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.138+0000 7faa667fc700 1 -- 172.21.15.78:0/3985609063 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faa6c0cefd0 con 0x7faa70102320 2024-03-16T11:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.138+0000 7faa74acd700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa5c0a2920 0x7faa5c0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faa70199e60 tx=0x7faa60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.142+0000 7faa667fc700 1 -- 172.21.15.78:0/3985609063 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa6c0042f0 con 0x7faa70102320 2024-03-16T11:05:14.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.335+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa70199bd0 con 0x7faa5c0a2920 2024-03-16T11:05:14.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.342+0000 7faa667fc700 1 -- 172.21.15.78:0/3985609063 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258715 (secure 0 0 0) 0x7faa70199bd0 con 0x7faa5c0a2920 2024-03-16T11:05:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.344+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa5c0a2920 msgr2=0x7faa5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.344+0000 7faa77532700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa5c0a2920 0x7faa5c0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faa70199e60 tx=0x7faa60009260 comp rx=0 tx=0).stop 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.344+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 msgr2=0x7faa7019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.344+0000 7faa77532700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 0x7faa7019e5d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00eb30 tx=0x7faa6c00ee40 comp rx=0 tx=0).stop 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 shutdown_connections 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faa5c0a2920 0x7faa5c0a4de0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa70102320 0x7faa7019e5d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa70108300 0x7faa701986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 --2- 172.21.15.78:0/3985609063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70102ce0 0x7faa7019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:14.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 >> 172.21.15.78:0/3985609063 conn(0x7faa70079400 msgr2=0x7faa70101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:14.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 shutdown_connections 2024-03-16T11:05:14.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:14.345+0000 7faa77532700 1 -- 172.21.15.78:0/3985609063 wait complete. 2024-03-16T11:05:14.349 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-16T11:05:15.133 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-03-16T11:05:13.983667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653515,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":556,"num_read_kb":392,"num_write":337,"num_write_kb":3045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375348,"kb_used_data":13644,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566476,"statfs":{"total":1151940427776,"available":1151556071424,"internally_reserved":0,"allocated":13971456,"data_stored":7606413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":34,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.881899"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106267+0000","last_change":"2024-03-16T11:04:00.295528+0000","last_active":"2024-03-16T11:05:03.106267+0000","last_peered":"2024-03-16T11:05:03.106267+0000","last_clean":"2024-03-16T11:05:03.106267+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:03.106267+0000","last_undegraded":"2024-03-16T11:05:03.106267+0000","last_fullsized":"2024-03-16T11:05:03.106267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:22.852488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105534+0000","last_change":"2024-03-16T11:04:00.289043+0000","last_active":"2024-03-16T11:05:03.105534+0000","last_peered":"2024-03-16T11:05:03.105534+0000","last_clean":"2024-03-16T11:05:03.105534+0000","last_became_active":"2024-03-16T11:04:00.288742+0000","last_became_peered":"2024-03-16T11:04:00.288742+0000","last_unstale":"2024-03-16T11:05:03.105534+0000","last_undegraded":"2024-03-16T11:05:03.105534+0000","last_fullsized":"2024-03-16T11:05:03.105534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:07:56.710499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105727+0000","last_change":"2024-03-16T11:04:00.293443+0000","last_active":"2024-03-16T11:05:03.105727+0000","last_peered":"2024-03-16T11:05:03.105727+0000","last_clean":"2024-03-16T11:05:03.105727+0000","last_became_active":"2024-03-16T11:04:00.293117+0000","last_became_peered":"2024-03-16T11:04:00.293117+0000","last_unstale":"2024-03-16T11:05:03.105727+0000","last_undegraded":"2024-03-16T11:05:03.105727+0000","last_fullsized":"2024-03-16T11:05:03.105727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:49.728721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542484+0000","last_change":"2024-03-16T11:04:00.292105+0000","last_active":"2024-03-16T11:05:03.542484+0000","last_peered":"2024-03-16T11:05:03.542484+0000","last_clean":"2024-03-16T11:05:03.542484+0000","last_became_active":"2024-03-16T11:04:00.291755+0000","last_became_peered":"2024-03-16T11:04:00.291755+0000","last_unstale":"2024-03-16T11:05:03.542484+0000","last_undegraded":"2024-03-16T11:05:03.542484+0000","last_fullsized":"2024-03-16T11:05:03.542484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:48:43.734854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108995+0000","last_change":"2024-03-16T11:04:00.282168+0000","last_active":"2024-03-16T11:05:03.108995+0000","last_peered":"2024-03-16T11:05:03.108995+0000","last_clean":"2024-03-16T11:05:03.108995+0000","last_became_active":"2024-03-16T11:04:00.281753+0000","last_became_peered":"2024-03-16T11:04:00.281753+0000","last_unstale":"2024-03-16T11:05:03.108995+0000","last_undegraded":"2024-03-16T11:05:03.108995+0000","last_fullsized":"2024-03-16T11:05:03.108995+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:13.223955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.111462+0000","last_change":"2024-03-16T11:05:03.111462+0000","last_active":"2024-03-16T11:05:03.111462+0000","last_peered":"2024-03-16T11:05:03.111462+0000","last_clean":"2024-03-16T11:05:03.111462+0000","last_became_active":"2024-03-16T11:05:03.111087+0000","last_became_peered":"2024-03-16T11:05:03.111087+0000","last_unstale":"2024-03-16T11:05:03.111462+0000","last_undegraded":"2024-03-16T11:05:03.111462+0000","last_fullsized":"2024-03-16T11:05:03.111462+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:09.339732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105242+0000","last_change":"2024-03-16T11:04:00.292446+0000","last_active":"2024-03-16T11:05:03.105242+0000","last_peered":"2024-03-16T11:05:03.105242+0000","last_clean":"2024-03-16T11:05:03.105242+0000","last_became_active":"2024-03-16T11:04:00.292079+0000","last_became_peered":"2024-03-16T11:04:00.292079+0000","last_unstale":"2024-03-16T11:05:03.105242+0000","last_undegraded":"2024-03-16T11:05:03.105242+0000","last_fullsized":"2024-03-16T11:05:03.105242+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.325945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106536+0000","last_change":"2024-03-16T11:04:00.294644+0000","last_active":"2024-03-16T11:05:03.106536+0000","last_peered":"2024-03-16T11:05:03.106536+0000","last_clean":"2024-03-16T11:05:03.106536+0000","last_became_active":"2024-03-16T11:04:00.294283+0000","last_became_peered":"2024-03-16T11:04:00.294283+0000","last_unstale":"2024-03-16T11:05:03.106536+0000","last_undegraded":"2024-03-16T11:05:03.106536+0000","last_fullsized":"2024-03-16T11:05:03.106536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:43.125823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101215+0000","last_change":"2024-03-16T11:04:00.302512+0000","last_active":"2024-03-16T11:05:03.101215+0000","last_peered":"2024-03-16T11:05:03.101215+0000","last_clean":"2024-03-16T11:05:03.101215+0000","last_became_active":"2024-03-16T11:04:00.301891+0000","last_became_peered":"2024-03-16T11:04:00.301891+0000","last_unstale":"2024-03-16T11:05:03.101215+0000","last_undegraded":"2024-03-16T11:05:03.101215+0000","last_fullsized":"2024-03-16T11:05:03.101215+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:15.112295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106507+0000","last_change":"2024-03-16T11:04:00.293327+0000","last_active":"2024-03-16T11:05:03.106507+0000","last_peered":"2024-03-16T11:05:03.106507+0000","last_clean":"2024-03-16T11:05:03.106507+0000","last_became_active":"2024-03-16T11:04:00.292971+0000","last_became_peered":"2024-03-16T11:04:00.292971+0000","last_unstale":"2024-03-16T11:05:03.106507+0000","last_undegraded":"2024-03-16T11:05:03.106507+0000","last_fullsized":"2024-03-16T11:05:03.106507+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:48:48.565861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105615+0000","last_change":"2024-03-16T11:04:00.298956+0000","last_active":"2024-03-16T11:05:03.105615+0000","last_peered":"2024-03-16T11:05:03.105615+0000","last_clean":"2024-03-16T11:05:03.105615+0000","last_became_active":"2024-03-16T11:04:00.298508+0000","last_became_peered":"2024-03-16T11:04:00.298508+0000","last_unstale":"2024-03-16T11:05:03.105615+0000","last_undegraded":"2024-03-16T11:05:03.105615+0000","last_fullsized":"2024-03-16T11:05:03.105615+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:25:46.576514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542435+0000","last_change":"2024-03-16T11:04:00.294453+0000","last_active":"2024-03-16T11:05:03.542435+0000","last_peered":"2024-03-16T11:05:03.542435+0000","last_clean":"2024-03-16T11:05:03.542435+0000","last_became_active":"2024-03-16T11:04:00.294084+0000","last_became_peered":"2024-03-16T11:04:00.294084+0000","last_unstale":"2024-03-16T11:05:03.542435+0000","last_undegraded":"2024-03-16T11:05:03.542435+0000","last_fullsized":"2024-03-16T11:05:03.542435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:12.924835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103504+0000","last_change":"2024-03-16T11:04:00.293149+0000","last_active":"2024-03-16T11:05:03.103504+0000","last_peered":"2024-03-16T11:05:03.103504+0000","last_clean":"2024-03-16T11:05:03.103504+0000","last_became_active":"2024-03-16T11:04:00.292839+0000","last_became_peered":"2024-03-16T11:04:00.292839+0000","last_unstale":"2024-03-16T11:05:03.103504+0000","last_undegraded":"2024-03-16T11:05:03.103504+0000","last_fullsized":"2024-03-16T11:05:03.103504+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:34:38.446918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105081+0000","last_change":"2024-03-16T11:04:00.285748+0000","last_active":"2024-03-16T11:05:03.105081+0000","last_peered":"2024-03-16T11:05:03.105081+0000","last_clean":"2024-03-16T11:05:03.105081+0000","last_became_active":"2024-03-16T11:04:00.285394+0000","last_became_peered":"2024-03-16T11:04:00.285394+0000","last_unstale":"2024-03-16T11:05:03.105081+0000","last_undegraded":"2024-03-16T11:05:03.105081+0000","last_fullsized":"2024-03-16T11:05:03.105081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:02.113271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.104818+0000","last_change":"2024-03-16T11:04:00.301783+0000","last_active":"2024-03-16T11:05:03.104818+0000","last_peered":"2024-03-16T11:05:03.104818+0000","last_clean":"2024-03-16T11:05:03.104818+0000","last_became_active":"2024-03-16T11:04:00.301313+0000","last_became_peered":"2024-03-16T11:04:00.301313+0000","last_unstale":"2024-03-16T11:05:03.104818+0000","last_undegraded":"2024-03-16T11:05:03.104818+0000","last_fullsized":"2024-03-16T11:05:03.104818+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:19:32.243602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.100914+0000","last_change":"2024-03-16T11:04:00.301375+0000","last_active":"2024-03-16T11:05:03.100914+0000","last_peered":"2024-03-16T11:05:03.100914+0000","last_clean":"2024-03-16T11:05:03.100914+0000","last_became_active":"2024-03-16T11:04:00.301013+0000","last_became_peered":"2024-03-16T11:04:00.301013+0000","last_unstale":"2024-03-16T11:05:03.100914+0000","last_undegraded":"2024-03-16T11:05:03.100914+0000","last_fullsized":"2024-03-16T11:05:03.100914+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:21:52.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106418+0000","last_change":"2024-03-16T11:04:00.296287+0000","last_active":"2024-03-16T11:05:03.106418+0000","last_peered":"2024-03-16T11:05:03.106418+0000","last_clean":"2024-03-16T11:05:03.106418+0000","last_became_active":"2024-03-16T11:04:00.295972+0000","last_became_peered":"2024-03-16T11:04:00.295972+0000","last_unstale":"2024-03-16T11:05:03.106418+0000","last_undegraded":"2024-03-16T11:05:03.106418+0000","last_fullsized":"2024-03-16T11:05:03.106418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:14.586052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102850+0000","last_change":"2024-03-16T11:04:00.295551+0000","last_active":"2024-03-16T11:05:03.102850+0000","last_peered":"2024-03-16T11:05:03.102850+0000","last_clean":"2024-03-16T11:05:03.102850+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:03.102850+0000","last_undegraded":"2024-03-16T11:05:03.102850+0000","last_fullsized":"2024-03-16T11:05:03.102850+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:53.495224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.549105+0000","last_change":"2024-03-16T11:04:00.290455+0000","last_active":"2024-03-16T11:05:03.549105+0000","last_peered":"2024-03-16T11:05:03.549105+0000","last_clean":"2024-03-16T11:05:03.549105+0000","last_became_active":"2024-03-16T11:04:00.290057+0000","last_became_peered":"2024-03-16T11:04:00.290057+0000","last_unstale":"2024-03-16T11:05:03.549105+0000","last_undegraded":"2024-03-16T11:05:03.549105+0000","last_fullsized":"2024-03-16T11:05:03.549105+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:57:43.677050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102676+0000","last_change":"2024-03-16T11:03:59.285059+0000","last_active":"2024-03-16T11:05:03.102676+0000","last_peered":"2024-03-16T11:05:03.102676+0000","last_clean":"2024-03-16T11:05:03.102676+0000","last_became_active":"2024-03-16T11:03:59.284745+0000","last_became_peered":"2024-03-16T11:03:59.284745+0000","last_unstale":"2024-03-16T11:05:03.102676+0000","last_undegraded":"2024-03-16T11:05:03.102676+0000","last_fullsized":"2024-03-16T11:05:03.102676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:12.516958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107404+0000","last_change":"2024-03-16T11:04:00.292933+0000","last_active":"2024-03-16T11:05:03.107404+0000","last_peered":"2024-03-16T11:05:03.107404+0000","last_clean":"2024-03-16T11:05:03.107404+0000","last_became_active":"2024-03-16T11:04:00.292637+0000","last_became_peered":"2024-03-16T11:04:00.292637+0000","last_unstale":"2024-03-16T11:05:03.107404+0000","last_undegraded":"2024-03-16T11:05:03.107404+0000","last_fullsized":"2024-03-16T11:05:03.107404+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:22.231735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107896+0000","last_change":"2024-03-16T11:03:59.285267+0000","last_active":"2024-03-16T11:05:03.107896+0000","last_peered":"2024-03-16T11:05:03.107896+0000","last_clean":"2024-03-16T11:05:03.107896+0000","last_became_active":"2024-03-16T11:03:59.284890+0000","last_became_peered":"2024-03-16T11:03:59.284890+0000","last_unstale":"2024-03-16T11:05:03.107896+0000","last_undegraded":"2024-03-16T11:05:03.107896+0000","last_fullsized":"2024-03-16T11:05:03.107896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:53.625612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105722+0000","last_change":"2024-03-16T11:04:00.292660+0000","last_active":"2024-03-16T11:05:03.105722+0000","last_peered":"2024-03-16T11:05:03.105722+0000","last_clean":"2024-03-16T11:05:03.105722+0000","last_became_active":"2024-03-16T11:04:00.292366+0000","last_became_peered":"2024-03-16T11:04:00.292366+0000","last_unstale":"2024-03-16T11:05:03.105722+0000","last_undegraded":"2024-03-16T11:05:03.105722+0000","last_fullsized":"2024-03-16T11:05:03.105722+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:36:44.331456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109296+0000","last_change":"2024-03-16T11:03:59.286370+0000","last_active":"2024-03-16T11:05:03.109296+0000","last_peered":"2024-03-16T11:05:03.109296+0000","last_clean":"2024-03-16T11:05:03.109296+0000","last_became_active":"2024-03-16T11:03:59.286065+0000","last_became_peered":"2024-03-16T11:03:59.286065+0000","last_unstale":"2024-03-16T11:05:03.109296+0000","last_undegraded":"2024-03-16T11:05:03.109296+0000","last_fullsized":"2024-03-16T11:05:03.109296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:20.126848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101112+0000","last_change":"2024-03-16T11:04:00.290864+0000","last_active":"2024-03-16T11:05:03.101112+0000","last_peered":"2024-03-16T11:05:03.101112+0000","last_clean":"2024-03-16T11:05:03.101112+0000","last_became_active":"2024-03-16T11:04:00.290537+0000","last_became_peered":"2024-03-16T11:04:00.290537+0000","last_unstale":"2024-03-16T11:05:03.101112+0000","last_undegraded":"2024-03-16T11:05:03.101112+0000","last_fullsized":"2024-03-16T11:05:03.101112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:52.991767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109155+0000","last_change":"2024-03-16T11:03:59.281663+0000","last_active":"2024-03-16T11:05:03.109155+0000","last_peered":"2024-03-16T11:05:03.109155+0000","last_clean":"2024-03-16T11:05:03.109155+0000","last_became_active":"2024-03-16T11:03:59.281328+0000","last_became_peered":"2024-03-16T11:03:59.281328+0000","last_unstale":"2024-03-16T11:05:03.109155+0000","last_undegraded":"2024-03-16T11:05:03.109155+0000","last_fullsized":"2024-03-16T11:05:03.109155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:44.511109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106586+0000","last_change":"2024-03-16T11:05:03.106586+0000","last_active":"2024-03-16T11:05:03.106586+0000","last_peered":"2024-03-16T11:05:03.106586+0000","last_clean":"2024-03-16T11:05:03.106586+0000","last_became_active":"2024-03-16T11:05:03.106233+0000","last_became_peered":"2024-03-16T11:05:03.106233+0000","last_unstale":"2024-03-16T11:05:03.106586+0000","last_undegraded":"2024-03-16T11:05:03.106586+0000","last_fullsized":"2024-03-16T11:05:03.106586+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:19:30.480318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:08.540308+0000","last_change":"2024-03-16T11:03:59.286300+0000","last_active":"2024-03-16T11:05:08.540308+0000","last_peered":"2024-03-16T11:05:08.540308+0000","last_clean":"2024-03-16T11:05:08.540308+0000","last_became_active":"2024-03-16T11:03:59.285764+0000","last_became_peered":"2024-03-16T11:03:59.285764+0000","last_unstale":"2024-03-16T11:05:08.540308+0000","last_undegraded":"2024-03-16T11:05:08.540308+0000","last_fullsized":"2024-03-16T11:05:08.540308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:49.084245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108999+0000","last_change":"2024-03-16T11:04:00.289152+0000","last_active":"2024-03-16T11:05:03.108999+0000","last_peered":"2024-03-16T11:05:03.108999+0000","last_clean":"2024-03-16T11:05:03.108999+0000","last_became_active":"2024-03-16T11:04:00.288837+0000","last_became_peered":"2024-03-16T11:04:00.288837+0000","last_unstale":"2024-03-16T11:05:03.108999+0000","last_undegraded":"2024-03-16T11:05:03.108999+0000","last_fullsized":"2024-03-16T11:05:03.108999+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:45:01.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106073+0000","last_change":"2024-03-16T11:03:59.281505+0000","last_active":"2024-03-16T11:05:03.106073+0000","last_peered":"2024-03-16T11:05:03.106073+0000","last_clean":"2024-03-16T11:05:03.106073+0000","last_became_active":"2024-03-16T11:03:59.281175+0000","last_became_peered":"2024-03-16T11:03:59.281175+0000","last_unstale":"2024-03-16T11:05:03.106073+0000","last_undegraded":"2024-03-16T11:05:03.106073+0000","last_fullsized":"2024-03-16T11:05:03.106073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:12.463914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103210+0000","last_change":"2024-03-16T11:04:00.297769+0000","last_active":"2024-03-16T11:05:03.103210+0000","last_peered":"2024-03-16T11:05:03.103210+0000","last_clean":"2024-03-16T11:05:03.103210+0000","last_became_active":"2024-03-16T11:04:00.297061+0000","last_became_peered":"2024-03-16T11:04:00.297061+0000","last_unstale":"2024-03-16T11:05:03.103210+0000","last_undegraded":"2024-03-16T11:05:03.103210+0000","last_fullsized":"2024-03-16T11:05:03.103210+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:42.594315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103214+0000","last_change":"2024-03-16T11:03:59.290260+0000","last_active":"2024-03-16T11:05:03.103214+0000","last_peered":"2024-03-16T11:05:03.103214+0000","last_clean":"2024-03-16T11:05:03.103214+0000","last_became_active":"2024-03-16T11:03:59.289843+0000","last_became_peered":"2024-03-16T11:03:59.289843+0000","last_unstale":"2024-03-16T11:05:03.103214+0000","last_undegraded":"2024-03-16T11:05:03.103214+0000","last_fullsized":"2024-03-16T11:05:03.103214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:22.483488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106580+0000","last_change":"2024-03-16T11:04:00.287873+0000","last_active":"2024-03-16T11:05:03.106580+0000","last_peered":"2024-03-16T11:05:03.106580+0000","last_clean":"2024-03-16T11:05:03.106580+0000","last_became_active":"2024-03-16T11:04:00.287002+0000","last_became_peered":"2024-03-16T11:04:00.287002+0000","last_unstale":"2024-03-16T11:05:03.106580+0000","last_undegraded":"2024-03-16T11:05:03.106580+0000","last_fullsized":"2024-03-16T11:05:03.106580+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:57.851726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102815+0000","last_change":"2024-03-16T11:03:59.284745+0000","last_active":"2024-03-16T11:05:03.102815+0000","last_peered":"2024-03-16T11:05:03.102815+0000","last_clean":"2024-03-16T11:05:03.102815+0000","last_became_active":"2024-03-16T11:03:59.284325+0000","last_became_peered":"2024-03-16T11:03:59.284325+0000","last_unstale":"2024-03-16T11:05:03.102815+0000","last_undegraded":"2024-03-16T11:05:03.102815+0000","last_fullsized":"2024-03-16T11:05:03.102815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.537787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102951+0000","last_change":"2024-03-16T11:04:00.290698+0000","last_active":"2024-03-16T11:05:03.102951+0000","last_peered":"2024-03-16T11:05:03.102951+0000","last_clean":"2024-03-16T11:05:03.102951+0000","last_became_active":"2024-03-16T11:04:00.290348+0000","last_became_peered":"2024-03-16T11:04:00.290348+0000","last_unstale":"2024-03-16T11:05:03.102951+0000","last_undegraded":"2024-03-16T11:05:03.102951+0000","last_fullsized":"2024-03-16T11:05:03.102951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:44.114305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:08.539818+0000","last_change":"2024-03-16T11:03:59.280705+0000","last_active":"2024-03-16T11:05:08.539818+0000","last_peered":"2024-03-16T11:05:08.539818+0000","last_clean":"2024-03-16T11:05:08.539818+0000","last_became_active":"2024-03-16T11:03:59.280372+0000","last_became_peered":"2024-03-16T11:03:59.280372+0000","last_unstale":"2024-03-16T11:05:08.539818+0000","last_undegraded":"2024-03-16T11:05:08.539818+0000","last_fullsized":"2024-03-16T11:05:08.539818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:54.187322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101124+0000","last_change":"2024-03-16T11:04:00.293867+0000","last_active":"2024-03-16T11:05:03.101124+0000","last_peered":"2024-03-16T11:05:03.101124+0000","last_clean":"2024-03-16T11:05:03.101124+0000","last_became_active":"2024-03-16T11:04:00.293330+0000","last_became_peered":"2024-03-16T11:04:00.293330+0000","last_unstale":"2024-03-16T11:05:03.101124+0000","last_undegraded":"2024-03-16T11:05:03.101124+0000","last_fullsized":"2024-03-16T11:05:03.101124+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:54.798346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105547+0000","last_change":"2024-03-16T11:03:59.284202+0000","last_active":"2024-03-16T11:05:03.105547+0000","last_peered":"2024-03-16T11:05:03.105547+0000","last_clean":"2024-03-16T11:05:03.105547+0000","last_became_active":"2024-03-16T11:03:59.283656+0000","last_became_peered":"2024-03-16T11:03:59.283656+0000","last_unstale":"2024-03-16T11:05:03.105547+0000","last_undegraded":"2024-03-16T11:05:03.105547+0000","last_fullsized":"2024-03-16T11:05:03.105547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:44.760307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.111443+0000","last_change":"2024-03-16T11:05:03.111443+0000","last_active":"2024-03-16T11:05:03.111443+0000","last_peered":"2024-03-16T11:05:03.111443+0000","last_clean":"2024-03-16T11:05:03.111443+0000","last_became_active":"2024-03-16T11:05:03.111027+0000","last_became_peered":"2024-03-16T11:05:03.111027+0000","last_unstale":"2024-03-16T11:05:03.111443+0000","last_undegraded":"2024-03-16T11:05:03.111443+0000","last_fullsized":"2024-03-16T11:05:03.111443+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:31.844283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103330+0000","last_change":"2024-03-16T11:03:59.288268+0000","last_active":"2024-03-16T11:05:03.103330+0000","last_peered":"2024-03-16T11:05:03.103330+0000","last_clean":"2024-03-16T11:05:03.103330+0000","last_became_active":"2024-03-16T11:03:59.287861+0000","last_became_peered":"2024-03-16T11:03:59.287861+0000","last_unstale":"2024-03-16T11:05:03.103330+0000","last_undegraded":"2024-03-16T11:05:03.103330+0000","last_fullsized":"2024-03-16T11:05:03.103330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:34:34.248408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101261+0000","last_change":"2024-03-16T11:04:00.306490+0000","last_active":"2024-03-16T11:05:03.101261+0000","last_peered":"2024-03-16T11:05:03.101261+0000","last_clean":"2024-03-16T11:05:03.101261+0000","last_became_active":"2024-03-16T11:04:00.305874+0000","last_became_peered":"2024-03-16T11:04:00.305874+0000","last_unstale":"2024-03-16T11:05:03.101261+0000","last_undegraded":"2024-03-16T11:05:03.101261+0000","last_fullsized":"2024-03-16T11:05:03.101261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'26","reported_seq":186,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:09.540920+0000","last_change":"2024-03-16T11:03:59.280714+0000","last_active":"2024-03-16T11:05:09.540920+0000","last_peered":"2024-03-16T11:05:09.540920+0000","last_clean":"2024-03-16T11:05:09.540920+0000","last_became_active":"2024-03-16T11:03:59.280398+0000","last_became_peered":"2024-03-16T11:03:59.280398+0000","last_unstale":"2024-03-16T11:05:09.540920+0000","last_undegraded":"2024-03-16T11:05:09.540920+0000","last_fullsized":"2024-03-16T11:05:09.540920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:34.256125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":141,"num_read_kb":127,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109217+0000","last_change":"2024-03-16T11:04:00.285669+0000","last_active":"2024-03-16T11:05:03.109217+0000","last_peered":"2024-03-16T11:05:03.109217+0000","last_clean":"2024-03-16T11:05:03.109217+0000","last_became_active":"2024-03-16T11:04:00.285337+0000","last_became_peered":"2024-03-16T11:04:00.285337+0000","last_unstale":"2024-03-16T11:05:03.109217+0000","last_undegraded":"2024-03-16T11:05:03.109217+0000","last_fullsized":"2024-03-16T11:05:03.109217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:36.925350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548506+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548506+0000","last_peered":"2024-03-16T11:05:03.548506+0000","last_clean":"2024-03-16T11:05:03.548506+0000","last_became_active":"2024-03-16T11:03:59.289357+0000","last_became_peered":"2024-03-16T11:03:59.289357+0000","last_unstale":"2024-03-16T11:05:03.548506+0000","last_undegraded":"2024-03-16T11:05:03.548506+0000","last_fullsized":"2024-03-16T11:05:03.548506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:26:35.860091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103313+0000","last_change":"2024-03-16T11:04:00.298488+0000","last_active":"2024-03-16T11:05:03.103313+0000","last_peered":"2024-03-16T11:05:03.103313+0000","last_clean":"2024-03-16T11:05:03.103313+0000","last_became_active":"2024-03-16T11:04:00.298202+0000","last_became_peered":"2024-03-16T11:04:00.298202+0000","last_unstale":"2024-03-16T11:05:03.103313+0000","last_undegraded":"2024-03-16T11:05:03.103313+0000","last_fullsized":"2024-03-16T11:05:03.103313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:48.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102961+0000","last_change":"2024-03-16T11:03:59.287066+0000","last_active":"2024-03-16T11:05:03.102961+0000","last_peered":"2024-03-16T11:05:03.102961+0000","last_clean":"2024-03-16T11:05:03.102961+0000","last_became_active":"2024-03-16T11:03:59.286731+0000","last_became_peered":"2024-03-16T11:03:59.286731+0000","last_unstale":"2024-03-16T11:05:03.102961+0000","last_undegraded":"2024-03-16T11:05:03.102961+0000","last_fullsized":"2024-03-16T11:05:03.102961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:51.443833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105488+0000","last_change":"2024-03-16T11:04:00.294450+0000","last_active":"2024-03-16T11:05:03.105488+0000","last_peered":"2024-03-16T11:05:03.105488+0000","last_clean":"2024-03-16T11:05:03.105488+0000","last_became_active":"2024-03-16T11:04:00.293769+0000","last_became_peered":"2024-03-16T11:04:00.293769+0000","last_unstale":"2024-03-16T11:05:03.105488+0000","last_undegraded":"2024-03-16T11:05:03.105488+0000","last_fullsized":"2024-03-16T11:05:03.105488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:02.755907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:07.546970+0000","last_change":"2024-03-16T11:03:59.286197+0000","last_active":"2024-03-16T11:05:07.546970+0000","last_peered":"2024-03-16T11:05:07.546970+0000","last_clean":"2024-03-16T11:05:07.546970+0000","last_became_active":"2024-03-16T11:03:59.285757+0000","last_became_peered":"2024-03-16T11:03:59.285757+0000","last_unstale":"2024-03-16T11:05:07.546970+0000","last_undegraded":"2024-03-16T11:05:07.546970+0000","last_fullsized":"2024-03-16T11:05:07.546970+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:47.868106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105321+0000","last_change":"2024-03-16T11:04:00.296709+0000","last_active":"2024-03-16T11:05:03.105321+0000","last_peered":"2024-03-16T11:05:03.105321+0000","last_clean":"2024-03-16T11:05:03.105321+0000","last_became_active":"2024-03-16T11:04:00.296372+0000","last_became_peered":"2024-03-16T11:04:00.296372+0000","last_unstale":"2024-03-16T11:05:03.105321+0000","last_undegraded":"2024-03-16T11:05:03.105321+0000","last_fullsized":"2024-03-16T11:05:03.105321+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:37.948866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108997+0000","last_change":"2024-03-16T11:03:59.283831+0000","last_active":"2024-03-16T11:05:03.108997+0000","last_peered":"2024-03-16T11:05:03.108997+0000","last_clean":"2024-03-16T11:05:03.108997+0000","last_became_active":"2024-03-16T11:03:59.283404+0000","last_became_peered":"2024-03-16T11:03:59.283404+0000","last_unstale":"2024-03-16T11:05:03.108997+0000","last_undegraded":"2024-03-16T11:05:03.108997+0000","last_fullsized":"2024-03-16T11:05:03.108997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:26:34.012597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102890+0000","last_change":"2024-03-16T11:04:00.299703+0000","last_active":"2024-03-16T11:05:03.102890+0000","last_peered":"2024-03-16T11:05:03.102890+0000","last_clean":"2024-03-16T11:05:03.102890+0000","last_became_active":"2024-03-16T11:04:00.299321+0000","last_became_peered":"2024-03-16T11:04:00.299321+0000","last_unstale":"2024-03-16T11:05:03.102890+0000","last_undegraded":"2024-03-16T11:05:03.102890+0000","last_fullsized":"2024-03-16T11:05:03.102890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.529281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109510+0000","last_change":"2024-03-16T11:03:59.277837+0000","last_active":"2024-03-16T11:05:03.109510+0000","last_peered":"2024-03-16T11:05:03.109510+0000","last_clean":"2024-03-16T11:05:03.109510+0000","last_became_active":"2024-03-16T11:03:59.277515+0000","last_became_peered":"2024-03-16T11:03:59.277515+0000","last_unstale":"2024-03-16T11:05:03.109510+0000","last_undegraded":"2024-03-16T11:05:03.109510+0000","last_fullsized":"2024-03-16T11:05:03.109510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:05.116990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106199+0000","last_change":"2024-03-16T11:04:00.292998+0000","last_active":"2024-03-16T11:05:03.106199+0000","last_peered":"2024-03-16T11:05:03.106199+0000","last_clean":"2024-03-16T11:05:03.106199+0000","last_became_active":"2024-03-16T11:04:00.292608+0000","last_became_peered":"2024-03-16T11:04:00.292608+0000","last_unstale":"2024-03-16T11:05:03.106199+0000","last_undegraded":"2024-03-16T11:05:03.106199+0000","last_fullsized":"2024-03-16T11:05:03.106199+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:41.996826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108505+0000","last_change":"2024-03-16T11:05:03.108505+0000","last_active":"2024-03-16T11:05:03.108505+0000","last_peered":"2024-03-16T11:05:03.108505+0000","last_clean":"2024-03-16T11:05:03.108505+0000","last_became_active":"2024-03-16T11:05:03.108132+0000","last_became_peered":"2024-03-16T11:05:03.108132+0000","last_unstale":"2024-03-16T11:05:03.108505+0000","last_undegraded":"2024-03-16T11:05:03.108505+0000","last_fullsized":"2024-03-16T11:05:03.108505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:50.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106434+0000","last_change":"2024-03-16T11:04:00.288897+0000","last_active":"2024-03-16T11:05:03.106434+0000","last_peered":"2024-03-16T11:05:03.106434+0000","last_clean":"2024-03-16T11:05:03.106434+0000","last_became_active":"2024-03-16T11:04:00.288493+0000","last_became_peered":"2024-03-16T11:04:00.288493+0000","last_unstale":"2024-03-16T11:05:03.106434+0000","last_undegraded":"2024-03-16T11:05:03.106434+0000","last_fullsized":"2024-03-16T11:05:03.106434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:31.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105515+0000","last_change":"2024-03-16T11:03:59.291595+0000","last_active":"2024-03-16T11:05:03.105515+0000","last_peered":"2024-03-16T11:05:03.105515+0000","last_clean":"2024-03-16T11:05:03.105515+0000","last_became_active":"2024-03-16T11:03:59.291159+0000","last_became_peered":"2024-03-16T11:03:59.291159+0000","last_unstale":"2024-03-16T11:05:03.105515+0000","last_undegraded":"2024-03-16T11:05:03.105515+0000","last_fullsized":"2024-03-16T11:05:03.105515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:03.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108818+0000","last_change":"2024-03-16T11:05:03.108818+0000","last_active":"2024-03-16T11:05:03.108818+0000","last_peered":"2024-03-16T11:05:03.108818+0000","last_clean":"2024-03-16T11:05:03.108818+0000","last_became_active":"2024-03-16T11:05:03.108479+0000","last_became_peered":"2024-03-16T11:05:03.108479+0000","last_unstale":"2024-03-16T11:05:03.108818+0000","last_undegraded":"2024-03-16T11:05:03.108818+0000","last_fullsized":"2024-03-16T11:05:03.108818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:10:11.329743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103022+0000","last_change":"2024-03-16T11:04:00.298400+0000","last_active":"2024-03-16T11:05:03.103022+0000","last_peered":"2024-03-16T11:05:03.103022+0000","last_clean":"2024-03-16T11:05:03.103022+0000","last_became_active":"2024-03-16T11:04:00.298026+0000","last_became_peered":"2024-03-16T11:04:00.298026+0000","last_unstale":"2024-03-16T11:05:03.103022+0000","last_undegraded":"2024-03-16T11:05:03.103022+0000","last_fullsized":"2024-03-16T11:05:03.103022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:08.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:12.551778+0000","last_change":"2024-03-16T11:03:59.287528+0000","last_active":"2024-03-16T11:05:12.551778+0000","last_peered":"2024-03-16T11:05:12.551778+0000","last_clean":"2024-03-16T11:05:12.551778+0000","last_became_active":"2024-03-16T11:03:59.287218+0000","last_became_peered":"2024-03-16T11:03:59.287218+0000","last_unstale":"2024-03-16T11:05:12.551778+0000","last_undegraded":"2024-03-16T11:05:12.551778+0000","last_fullsized":"2024-03-16T11:05:12.551778+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:19.803233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106614+0000","last_change":"2024-03-16T11:05:03.106099+0000","last_active":"2024-03-16T11:05:03.106614+0000","last_peered":"2024-03-16T11:05:03.106614+0000","last_clean":"2024-03-16T11:05:03.106614+0000","last_became_active":"2024-03-16T11:05:03.105412+0000","last_became_peered":"2024-03-16T11:05:03.105412+0000","last_unstale":"2024-03-16T11:05:03.106614+0000","last_undegraded":"2024-03-16T11:05:03.106614+0000","last_fullsized":"2024-03-16T11:05:03.106614+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:29:33.213676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.541779+0000","last_change":"2024-03-16T11:03:59.281251+0000","last_active":"2024-03-16T11:05:03.541779+0000","last_peered":"2024-03-16T11:05:03.541779+0000","last_clean":"2024-03-16T11:05:03.541779+0000","last_became_active":"2024-03-16T11:03:59.280830+0000","last_became_peered":"2024-03-16T11:03:59.280830+0000","last_unstale":"2024-03-16T11:05:03.541779+0000","last_undegraded":"2024-03-16T11:05:03.541779+0000","last_fullsized":"2024-03-16T11:05:03.541779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:03:04.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105426+0000","last_change":"2024-03-16T11:02:48.497749+0000","last_active":"2024-03-16T11:05:03.105426+0000","last_peered":"2024-03-16T11:05:03.105426+0000","last_clean":"2024-03-16T11:05:03.105426+0000","last_became_active":"2024-03-16T11:02:48.497157+0000","last_became_peered":"2024-03-16T11:02:48.497157+0000","last_unstale":"2024-03-16T11:05:03.105426+0000","last_undegraded":"2024-03-16T11:05:03.105426+0000","last_fullsized":"2024-03-16T11:05:03.105426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_clean_scrub_stamp":"2024-03-16T10:59:26.309639+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:04.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102952+0000","last_change":"2024-03-16T11:04:00.300821+0000","last_active":"2024-03-16T11:05:03.102952+0000","last_peered":"2024-03-16T11:05:03.102952+0000","last_clean":"2024-03-16T11:05:03.102952+0000","last_became_active":"2024-03-16T11:04:00.300510+0000","last_became_peered":"2024-03-16T11:04:00.300510+0000","last_unstale":"2024-03-16T11:05:03.102952+0000","last_undegraded":"2024-03-16T11:05:03.102952+0000","last_fullsized":"2024-03-16T11:05:03.102952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:57.035167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105571+0000","last_change":"2024-03-16T11:03:59.283609+0000","last_active":"2024-03-16T11:05:03.105571+0000","last_peered":"2024-03-16T11:05:03.105571+0000","last_clean":"2024-03-16T11:05:03.105571+0000","last_became_active":"2024-03-16T11:03:59.283227+0000","last_became_peered":"2024-03-16T11:03:59.283227+0000","last_unstale":"2024-03-16T11:05:03.105571+0000","last_undegraded":"2024-03-16T11:05:03.105571+0000","last_fullsized":"2024-03-16T11:05:03.105571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:18.904202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.112457+0000","last_change":"2024-03-16T11:05:03.112071+0000","last_active":"2024-03-16T11:05:03.112457+0000","last_peered":"2024-03-16T11:05:03.112457+0000","last_clean":"2024-03-16T11:05:03.112457+0000","last_became_active":"2024-03-16T11:05:03.108434+0000","last_became_peered":"2024-03-16T11:05:03.108434+0000","last_unstale":"2024-03-16T11:05:03.112457+0000","last_undegraded":"2024-03-16T11:05:03.112457+0000","last_fullsized":"2024-03-16T11:05:03.112457+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:24.758886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103481+0000","last_change":"2024-03-16T11:04:00.285134+0000","last_active":"2024-03-16T11:05:03.103481+0000","last_peered":"2024-03-16T11:05:03.103481+0000","last_clean":"2024-03-16T11:05:03.103481+0000","last_became_active":"2024-03-16T11:04:00.284705+0000","last_became_peered":"2024-03-16T11:04:00.284705+0000","last_unstale":"2024-03-16T11:05:03.103481+0000","last_undegraded":"2024-03-16T11:05:03.103481+0000","last_fullsized":"2024-03-16T11:05:03.103481+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:38.637970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102736+0000","last_change":"2024-03-16T11:03:59.289999+0000","last_active":"2024-03-16T11:05:03.102736+0000","last_peered":"2024-03-16T11:05:03.102736+0000","last_clean":"2024-03-16T11:05:03.102736+0000","last_became_active":"2024-03-16T11:03:59.289694+0000","last_became_peered":"2024-03-16T11:03:59.289694+0000","last_unstale":"2024-03-16T11:05:03.102736+0000","last_undegraded":"2024-03-16T11:05:03.102736+0000","last_fullsized":"2024-03-16T11:05:03.102736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:16.037387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107698+0000","last_change":"2024-03-16T11:04:00.298148+0000","last_active":"2024-03-16T11:05:03.107698+0000","last_peered":"2024-03-16T11:05:03.107698+0000","last_clean":"2024-03-16T11:05:03.107698+0000","last_became_active":"2024-03-16T11:04:00.297822+0000","last_became_peered":"2024-03-16T11:04:00.297822+0000","last_unstale":"2024-03-16T11:05:03.107698+0000","last_undegraded":"2024-03-16T11:05:03.107698+0000","last_fullsized":"2024-03-16T11:05:03.107698+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:09:28.645317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542005+0000","last_change":"2024-03-16T11:03:59.292207+0000","last_active":"2024-03-16T11:05:03.542005+0000","last_peered":"2024-03-16T11:05:03.542005+0000","last_clean":"2024-03-16T11:05:03.542005+0000","last_became_active":"2024-03-16T11:03:59.291859+0000","last_became_peered":"2024-03-16T11:03:59.291859+0000","last_unstale":"2024-03-16T11:05:03.542005+0000","last_undegraded":"2024-03-16T11:05:03.542005+0000","last_fullsized":"2024-03-16T11:05:03.542005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:55.365720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105643+0000","last_change":"2024-03-16T11:04:00.291261+0000","last_active":"2024-03-16T11:05:03.105643+0000","last_peered":"2024-03-16T11:05:03.105643+0000","last_clean":"2024-03-16T11:05:03.105643+0000","last_became_active":"2024-03-16T11:04:00.290832+0000","last_became_peered":"2024-03-16T11:04:00.290832+0000","last_unstale":"2024-03-16T11:05:03.105643+0000","last_undegraded":"2024-03-16T11:05:03.105643+0000","last_fullsized":"2024-03-16T11:05:03.105643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:10.418808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.100986+0000","last_change":"2024-03-16T11:03:59.288014+0000","last_active":"2024-03-16T11:05:03.100986+0000","last_peered":"2024-03-16T11:05:03.100986+0000","last_clean":"2024-03-16T11:05:03.100986+0000","last_became_active":"2024-03-16T11:03:59.287679+0000","last_became_peered":"2024-03-16T11:03:59.287679+0000","last_unstale":"2024-03-16T11:05:03.100986+0000","last_undegraded":"2024-03-16T11:05:03.100986+0000","last_fullsized":"2024-03-16T11:05:03.100986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:16:39.156224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108907+0000","last_change":"2024-03-16T11:04:00.288047+0000","last_active":"2024-03-16T11:05:03.108907+0000","last_peered":"2024-03-16T11:05:03.108907+0000","last_clean":"2024-03-16T11:05:03.108907+0000","last_became_active":"2024-03-16T11:04:00.287619+0000","last_became_peered":"2024-03-16T11:04:00.287619+0000","last_unstale":"2024-03-16T11:05:03.108907+0000","last_undegraded":"2024-03-16T11:05:03.108907+0000","last_fullsized":"2024-03-16T11:05:03.108907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:43.608349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106759+0000","last_change":"2024-03-16T11:03:59.288439+0000","last_active":"2024-03-16T11:05:03.106759+0000","last_peered":"2024-03-16T11:05:03.106759+0000","last_clean":"2024-03-16T11:05:03.106759+0000","last_became_active":"2024-03-16T11:03:59.288122+0000","last_became_peered":"2024-03-16T11:03:59.288122+0000","last_unstale":"2024-03-16T11:05:03.106759+0000","last_undegraded":"2024-03-16T11:05:03.106759+0000","last_fullsized":"2024-03-16T11:05:03.106759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:50:25.499608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105759+0000","last_change":"2024-03-16T11:04:00.290522+0000","last_active":"2024-03-16T11:05:03.105759+0000","last_peered":"2024-03-16T11:05:03.105759+0000","last_clean":"2024-03-16T11:05:03.105759+0000","last_became_active":"2024-03-16T11:04:00.290175+0000","last_became_peered":"2024-03-16T11:04:00.290175+0000","last_unstale":"2024-03-16T11:05:03.105759+0000","last_undegraded":"2024-03-16T11:05:03.105759+0000","last_fullsized":"2024-03-16T11:05:03.105759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:34.113278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548452+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548452+0000","last_peered":"2024-03-16T11:05:03.548452+0000","last_clean":"2024-03-16T11:05:03.548452+0000","last_became_active":"2024-03-16T11:03:59.289367+0000","last_became_peered":"2024-03-16T11:03:59.289367+0000","last_unstale":"2024-03-16T11:05:03.548452+0000","last_undegraded":"2024-03-16T11:05:03.548452+0000","last_fullsized":"2024-03-16T11:05:03.548452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:42:48.627116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107105+0000","last_change":"2024-03-16T11:04:00.287685+0000","last_active":"2024-03-16T11:05:03.107105+0000","last_peered":"2024-03-16T11:05:03.107105+0000","last_clean":"2024-03-16T11:05:03.107105+0000","last_became_active":"2024-03-16T11:04:00.287367+0000","last_became_peered":"2024-03-16T11:04:00.287367+0000","last_unstale":"2024-03-16T11:05:03.107105+0000","last_undegraded":"2024-03-16T11:05:03.107105+0000","last_fullsized":"2024-03-16T11:05:03.107105+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:25:52.741651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108923+0000","last_change":"2024-03-16T11:03:59.286293+0000","last_active":"2024-03-16T11:05:03.108923+0000","last_peered":"2024-03-16T11:05:03.108923+0000","last_clean":"2024-03-16T11:05:03.108923+0000","last_became_active":"2024-03-16T11:03:59.285817+0000","last_became_peered":"2024-03-16T11:03:59.285817+0000","last_unstale":"2024-03-16T11:05:03.108923+0000","last_undegraded":"2024-03-16T11:05:03.108923+0000","last_fullsized":"2024-03-16T11:05:03.108923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:38:12.093762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105889+0000","last_change":"2024-03-16T11:04:00.299391+0000","last_active":"2024-03-16T11:05:03.105889+0000","last_peered":"2024-03-16T11:05:03.105889+0000","last_clean":"2024-03-16T11:05:03.105889+0000","last_became_active":"2024-03-16T11:04:00.298931+0000","last_became_peered":"2024-03-16T11:04:00.298931+0000","last_unstale":"2024-03-16T11:05:03.105889+0000","last_undegraded":"2024-03-16T11:05:03.105889+0000","last_fullsized":"2024-03-16T11:05:03.105889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:12:11.900154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548534+0000","last_change":"2024-03-16T11:03:59.292319+0000","last_active":"2024-03-16T11:05:03.548534+0000","last_peered":"2024-03-16T11:05:03.548534+0000","last_clean":"2024-03-16T11:05:03.548534+0000","last_became_active":"2024-03-16T11:03:59.291953+0000","last_became_peered":"2024-03-16T11:03:59.291953+0000","last_unstale":"2024-03-16T11:05:03.548534+0000","last_undegraded":"2024-03-16T11:05:03.548534+0000","last_fullsized":"2024-03-16T11:05:03.548534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:41:04.615726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105749+0000","last_change":"2024-03-16T11:04:00.296544+0000","last_active":"2024-03-16T11:05:03.105749+0000","last_peered":"2024-03-16T11:05:03.105749+0000","last_clean":"2024-03-16T11:05:03.105749+0000","last_became_active":"2024-03-16T11:04:00.296078+0000","last_became_peered":"2024-03-16T11:04:00.296078+0000","last_unstale":"2024-03-16T11:05:03.105749+0000","last_undegraded":"2024-03-16T11:05:03.105749+0000","last_fullsized":"2024-03-16T11:05:03.105749+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:56:22.846734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103250+0000","last_change":"2024-03-16T11:03:59.287172+0000","last_active":"2024-03-16T11:05:03.103250+0000","last_peered":"2024-03-16T11:05:03.103250+0000","last_clean":"2024-03-16T11:05:03.103250+0000","last_became_active":"2024-03-16T11:03:59.286785+0000","last_became_peered":"2024-03-16T11:03:59.286785+0000","last_unstale":"2024-03-16T11:05:03.103250+0000","last_undegraded":"2024-03-16T11:05:03.103250+0000","last_fullsized":"2024-03-16T11:05:03.103250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:17:55.584618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.111534+0000","last_change":"2024-03-16T11:05:03.111534+0000","last_active":"2024-03-16T11:05:03.111534+0000","last_peered":"2024-03-16T11:05:03.111534+0000","last_clean":"2024-03-16T11:05:03.111534+0000","last_became_active":"2024-03-16T11:05:03.107517+0000","last_became_peered":"2024-03-16T11:05:03.107517+0000","last_unstale":"2024-03-16T11:05:03.111534+0000","last_undegraded":"2024-03-16T11:05:03.111534+0000","last_fullsized":"2024-03-16T11:05:03.111534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:07:51.828833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103117+0000","last_change":"2024-03-16T11:03:59.288822+0000","last_active":"2024-03-16T11:05:03.103117+0000","last_peered":"2024-03-16T11:05:03.103117+0000","last_clean":"2024-03-16T11:05:03.103117+0000","last_became_active":"2024-03-16T11:03:59.288516+0000","last_became_peered":"2024-03-16T11:03:59.288516+0000","last_unstale":"2024-03-16T11:05:03.103117+0000","last_undegraded":"2024-03-16T11:05:03.103117+0000","last_fullsized":"2024-03-16T11:05:03.103117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:32:38.862476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105721+0000","last_change":"2024-03-16T11:04:00.296117+0000","last_active":"2024-03-16T11:05:03.105721+0000","last_peered":"2024-03-16T11:05:03.105721+0000","last_clean":"2024-03-16T11:05:03.105721+0000","last_became_active":"2024-03-16T11:04:00.295664+0000","last_became_peered":"2024-03-16T11:04:00.295664+0000","last_unstale":"2024-03-16T11:05:03.105721+0000","last_undegraded":"2024-03-16T11:05:03.105721+0000","last_fullsized":"2024-03-16T11:05:03.105721+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.448616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105812+0000","last_change":"2024-03-16T11:04:00.295035+0000","last_active":"2024-03-16T11:05:03.105812+0000","last_peered":"2024-03-16T11:05:03.105812+0000","last_clean":"2024-03-16T11:05:03.105812+0000","last_became_active":"2024-03-16T11:04:00.294689+0000","last_became_peered":"2024-03-16T11:04:00.294689+0000","last_unstale":"2024-03-16T11:05:03.105812+0000","last_undegraded":"2024-03-16T11:05:03.105812+0000","last_fullsized":"2024-03-16T11:05:03.105812+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:55.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548695+0000","last_change":"2024-03-16T11:04:00.292831+0000","last_active":"2024-03-16T11:05:03.548695+0000","last_peered":"2024-03-16T11:05:03.548695+0000","last_clean":"2024-03-16T11:05:03.548695+0000","last_became_active":"2024-03-16T11:04:00.292387+0000","last_became_peered":"2024-03-16T11:04:00.292387+0000","last_unstale":"2024-03-16T11:05:03.548695+0000","last_undegraded":"2024-03-16T11:05:03.548695+0000","last_fullsized":"2024-03-16T11:05:03.548695+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:14:53.273535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548822+0000","last_change":"2024-03-16T11:04:00.291267+0000","last_active":"2024-03-16T11:05:03.548822+0000","last_peered":"2024-03-16T11:05:03.548822+0000","last_clean":"2024-03-16T11:05:03.548822+0000","last_became_active":"2024-03-16T11:04:00.290911+0000","last_became_peered":"2024-03-16T11:04:00.290911+0000","last_unstale":"2024-03-16T11:05:03.548822+0000","last_undegraded":"2024-03-16T11:05:03.548822+0000","last_fullsized":"2024-03-16T11:05:03.548822+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:35.285167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105324+0000","last_change":"2024-03-16T11:04:00.300470+0000","last_active":"2024-03-16T11:05:03.105324+0000","last_peered":"2024-03-16T11:05:03.105324+0000","last_clean":"2024-03-16T11:05:03.105324+0000","last_became_active":"2024-03-16T11:04:00.300107+0000","last_became_peered":"2024-03-16T11:04:00.300107+0000","last_unstale":"2024-03-16T11:05:03.105324+0000","last_undegraded":"2024-03-16T11:05:03.105324+0000","last_fullsized":"2024-03-16T11:05:03.105324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:18.718749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103549+0000","last_change":"2024-03-16T11:04:00.300857+0000","last_active":"2024-03-16T11:05:03.103549+0000","last_peered":"2024-03-16T11:05:03.103549+0000","last_clean":"2024-03-16T11:05:03.103549+0000","last_became_active":"2024-03-16T11:04:00.300551+0000","last_became_peered":"2024-03-16T11:04:00.300551+0000","last_unstale":"2024-03-16T11:05:03.103549+0000","last_undegraded":"2024-03-16T11:05:03.103549+0000","last_fullsized":"2024-03-16T11:05:03.103549+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:24.362561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.110781+0000","last_change":"2024-03-16T11:05:03.110781+0000","last_active":"2024-03-16T11:05:03.110781+0000","last_peered":"2024-03-16T11:05:03.110781+0000","last_clean":"2024-03-16T11:05:03.110781+0000","last_became_active":"2024-03-16T11:05:03.110375+0000","last_became_peered":"2024-03-16T11:05:03.110375+0000","last_unstale":"2024-03-16T11:05:03.110781+0000","last_undegraded":"2024-03-16T11:05:03.110781+0000","last_fullsized":"2024-03-16T11:05:03.110781+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:28.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.549286+0000","last_change":"2024-03-16T11:04:00.294943+0000","last_active":"2024-03-16T11:05:03.549286+0000","last_peered":"2024-03-16T11:05:03.549286+0000","last_clean":"2024-03-16T11:05:03.549286+0000","last_became_active":"2024-03-16T11:04:00.294637+0000","last_became_peered":"2024-03-16T11:04:00.294637+0000","last_unstale":"2024-03-16T11:05:03.549286+0000","last_undegraded":"2024-03-16T11:05:03.549286+0000","last_fullsized":"2024-03-16T11:05:03.549286+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:03:09.716753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107606+0000","last_change":"2024-03-16T11:04:00.300737+0000","last_active":"2024-03-16T11:05:03.107606+0000","last_peered":"2024-03-16T11:05:03.107606+0000","last_clean":"2024-03-16T11:05:03.107606+0000","last_became_active":"2024-03-16T11:04:00.300410+0000","last_became_peered":"2024-03-16T11:04:00.300410+0000","last_unstale":"2024-03-16T11:05:03.107606+0000","last_undegraded":"2024-03-16T11:05:03.107606+0000","last_fullsized":"2024-03-16T11:05:03.107606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:18.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102837+0000","last_change":"2024-03-16T11:04:00.292958+0000","last_active":"2024-03-16T11:05:03.102837+0000","last_peered":"2024-03-16T11:05:03.102837+0000","last_clean":"2024-03-16T11:05:03.102837+0000","last_became_active":"2024-03-16T11:04:00.292599+0000","last_became_peered":"2024-03-16T11:04:00.292599+0000","last_unstale":"2024-03-16T11:05:03.102837+0000","last_undegraded":"2024-03-16T11:05:03.102837+0000","last_fullsized":"2024-03-16T11:05:03.102837+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:46.370597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106664+0000","last_change":"2024-03-16T11:04:00.291381+0000","last_active":"2024-03-16T11:05:03.106664+0000","last_peered":"2024-03-16T11:05:03.106664+0000","last_clean":"2024-03-16T11:05:03.106664+0000","last_became_active":"2024-03-16T11:04:00.291046+0000","last_became_peered":"2024-03-16T11:04:00.291046+0000","last_unstale":"2024-03-16T11:05:03.106664+0000","last_undegraded":"2024-03-16T11:05:03.106664+0000","last_fullsized":"2024-03-16T11:05:03.106664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:08.185733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101021+0000","last_change":"2024-03-16T11:04:00.301381+0000","last_active":"2024-03-16T11:05:03.101021+0000","last_peered":"2024-03-16T11:05:03.101021+0000","last_clean":"2024-03-16T11:05:03.101021+0000","last_became_active":"2024-03-16T11:04:00.300990+0000","last_became_peered":"2024-03-16T11:04:00.300990+0000","last_unstale":"2024-03-16T11:05:03.101021+0000","last_undegraded":"2024-03-16T11:05:03.101021+0000","last_fullsized":"2024-03-16T11:05:03.101021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:07:43.775066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103609+0000","last_change":"2024-03-16T11:04:00.298220+0000","last_active":"2024-03-16T11:05:03.103609+0000","last_peered":"2024-03-16T11:05:03.103609+0000","last_clean":"2024-03-16T11:05:03.103609+0000","last_became_active":"2024-03-16T11:04:00.297776+0000","last_became_peered":"2024-03-16T11:04:00.297776+0000","last_unstale":"2024-03-16T11:05:03.103609+0000","last_undegraded":"2024-03-16T11:05:03.103609+0000","last_fullsized":"2024-03-16T11:05:03.103609+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:40.387309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106051+0000","last_change":"2024-03-16T11:04:00.292988+0000","last_active":"2024-03-16T11:05:03.106051+0000","last_peered":"2024-03-16T11:05:03.106051+0000","last_clean":"2024-03-16T11:05:03.106051+0000","last_became_active":"2024-03-16T11:04:00.292655+0000","last_became_peered":"2024-03-16T11:04:00.292655+0000","last_unstale":"2024-03-16T11:05:03.106051+0000","last_undegraded":"2024-03-16T11:05:03.106051+0000","last_fullsized":"2024-03-16T11:05:03.106051+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:33:43.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62913,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":430,"num_read_kb":283,"num_write":154,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":188739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285595,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":471093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547233,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32824,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712328,"statfs":{"total":95995035648,"available":95961423872,"internally_reserved":0,"allocated":1679360,"data_stored":1121855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743463,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":471307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005099,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":471205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266736,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1063157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430262,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":470813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":529401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1007616,"data_stored":471967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":530751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":471007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888125,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":471524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691899,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1601536,"data_stored":1062333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:05:15.137 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T11:05:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:15 smithi098 ceph-mon[35848]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 12 B/s, 0 objects/s recovering 2024-03-16T11:05:15.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:15 smithi078 ceph-mon[32235]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 12 B/s, 0 objects/s recovering 2024-03-16T11:05:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:15 smithi153 ceph-mon[35521]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 12 B/s, 0 objects/s recovering 2024-03-16T11:05:16.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:16 smithi098 ceph-mon[35848]: from='client.24799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:16.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:16 smithi078 ceph-mon[32235]: from='client.24799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:16.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:16 smithi153 ceph-mon[35521]: from='client.24799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:16.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 -- 172.21.15.78:0/2819632410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7424101690 msgr2=0x7f742410f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:16.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 --2- 172.21.15.78:0/2819632410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7424101690 0x7f742410f220 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f741c00b0d0 tx=0x7f741c00b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:16.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 -- 172.21.15.78:0/2819632410 shutdown_connections 2024-03-16T11:05:16.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 --2- 172.21.15.78:0/2819632410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74241073a0 0x7f7424107780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:16.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 --2- 172.21.15.78:0/2819632410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7424101690 0x7f742410f220 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:16.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 --2- 172.21.15.78:0/2819632410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424100cd0 0x7f7424101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:16.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 -- 172.21.15.78:0/2819632410 >> 172.21.15.78:0/2819632410 conn(0x7f7424076520 msgr2=0x7f7424076930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 -- 172.21.15.78:0/2819632410 shutdown_connections 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 -- 172.21.15.78:0/2819632410 wait complete. 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 Processor -- start 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.997+0000 7f742b051700 1 -- start start 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f742b051700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424100cd0 0x7f742419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f742b051700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424101690 0x7f742419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f742b051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74241073a0 0x7f74241987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f742b051700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f74241124f0 con 0x7f74241073a0 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f742b051700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7424112370 con 0x7f7424101690 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f742b051700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7424112670 con 0x7f7424100cd0 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f74295ee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74241073a0 0x7f74241987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7423fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424101690 0x7f742419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:17.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f74295ee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74241073a0 0x7f74241987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42658/0 (socket says 172.21.15.78:42658) 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f74295ee700 1 -- 172.21.15.78:0/3306959202 learned_addr learned my addr 172.21.15.78:0/3306959202 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7428ded700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424100cd0 0x7f742419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7423fff700 1 -- 172.21.15.78:0/3306959202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424100cd0 msgr2=0x7f742419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7423fff700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424100cd0 0x7f742419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7423fff700 1 -- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74241073a0 msgr2=0x7f74241987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7423fff700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74241073a0 0x7f74241987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7423fff700 1 -- 172.21.15.78:0/3306959202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7424198fd0 con 0x7f7424101690 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f74295ee700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74241073a0 0x7f74241987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7423fff700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424101690 0x7f742419eb60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f741000e9f0 tx=0x7f741000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7421ffb700 1 -- 172.21.15.78:0/3306959202 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741000c9c0 con 0x7f7424101690 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7421ffb700 1 -- 172.21.15.78:0/3306959202 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f741000cb60 con 0x7f7424101690 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74241992c0 con 0x7f7424101690 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.998+0000 7f7421ffb700 1 -- 172.21.15.78:0/3306959202 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7410018680 con 0x7f7424101690 2024-03-16T11:05:17.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.999+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74241a5480 con 0x7f7424101690 2024-03-16T11:05:17.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:16.999+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742410a8c0 con 0x7f7424101690 2024-03-16T11:05:17.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.001+0000 7f7421ffb700 1 -- 172.21.15.78:0/3306959202 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7410010430 con 0x7f7424101690 2024-03-16T11:05:17.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.001+0000 7f7421ffb700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74140a2920 0x7f74140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:17.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.001+0000 7f7428ded700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74140a2920 0x7f74140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:17.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.001+0000 7f7421ffb700 1 -- 172.21.15.78:0/3306959202 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f74100cf000 con 0x7f7424101690 2024-03-16T11:05:17.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.002+0000 7f7428ded700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74140a2920 0x7f74140a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7418009be0 tx=0x7f7418009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:17.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.005+0000 7f7421ffb700 1 -- 172.21.15.78:0/3306959202 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f741009bf40 con 0x7f7424101690 2024-03-16T11:05:17.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.199+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7424062710 con 0x7f74140a2920 2024-03-16T11:05:17.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.206+0000 7f7421ffb700 1 -- 172.21.15.78:0/3306959202 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258714 (secure 0 0 0) 0x7f7424062710 con 0x7f74140a2920 2024-03-16T11:05:17.210 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74140a2920 msgr2=0x7f74140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74140a2920 0x7f74140a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7418009be0 tx=0x7f7418009260 comp rx=0 tx=0).stop 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424101690 msgr2=0x7f742419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424101690 0x7f742419eb60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f741000e9f0 tx=0x7f741000ed00 comp rx=0 tx=0).stop 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 shutdown_connections 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f74140a2920 0x7f74140a4de0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424100cd0 0x7f742419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74241073a0 0x7f74241987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 --2- 172.21.15.78:0/3306959202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7424101690 0x7f742419eb60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:17.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 >> 172.21.15.78:0/3306959202 conn(0x7f7424076520 msgr2=0x7f74240fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:17.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 shutdown_connections 2024-03-16T11:05:17.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:17.209+0000 7f742b051700 1 -- 172.21.15.78:0/3306959202 wait complete. 2024-03-16T11:05:17.213 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-16T11:05:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:17 smithi098 ceph-mon[35848]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T11:05:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:17 smithi153 ceph-mon[35521]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T11:05:17.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:17 smithi078 ceph-mon[32235]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T11:05:17.920 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-03-16T11:05:15.984141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653515,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":581,"num_read_kb":402,"num_write":337,"num_write_kb":3045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375348,"kb_used_data":13644,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566476,"statfs":{"total":1151940427776,"available":1151556071424,"internally_reserved":0,"allocated":13971456,"data_stored":7606413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":84,"num_read_kb":39,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002802"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106267+0000","last_change":"2024-03-16T11:04:00.295528+0000","last_active":"2024-03-16T11:05:03.106267+0000","last_peered":"2024-03-16T11:05:03.106267+0000","last_clean":"2024-03-16T11:05:03.106267+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:03.106267+0000","last_undegraded":"2024-03-16T11:05:03.106267+0000","last_fullsized":"2024-03-16T11:05:03.106267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:22.852488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105534+0000","last_change":"2024-03-16T11:04:00.289043+0000","last_active":"2024-03-16T11:05:03.105534+0000","last_peered":"2024-03-16T11:05:03.105534+0000","last_clean":"2024-03-16T11:05:03.105534+0000","last_became_active":"2024-03-16T11:04:00.288742+0000","last_became_peered":"2024-03-16T11:04:00.288742+0000","last_unstale":"2024-03-16T11:05:03.105534+0000","last_undegraded":"2024-03-16T11:05:03.105534+0000","last_fullsized":"2024-03-16T11:05:03.105534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:07:56.710499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105727+0000","last_change":"2024-03-16T11:04:00.293443+0000","last_active":"2024-03-16T11:05:03.105727+0000","last_peered":"2024-03-16T11:05:03.105727+0000","last_clean":"2024-03-16T11:05:03.105727+0000","last_became_active":"2024-03-16T11:04:00.293117+0000","last_became_peered":"2024-03-16T11:04:00.293117+0000","last_unstale":"2024-03-16T11:05:03.105727+0000","last_undegraded":"2024-03-16T11:05:03.105727+0000","last_fullsized":"2024-03-16T11:05:03.105727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:49.728721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542484+0000","last_change":"2024-03-16T11:04:00.292105+0000","last_active":"2024-03-16T11:05:03.542484+0000","last_peered":"2024-03-16T11:05:03.542484+0000","last_clean":"2024-03-16T11:05:03.542484+0000","last_became_active":"2024-03-16T11:04:00.291755+0000","last_became_peered":"2024-03-16T11:04:00.291755+0000","last_unstale":"2024-03-16T11:05:03.542484+0000","last_undegraded":"2024-03-16T11:05:03.542484+0000","last_fullsized":"2024-03-16T11:05:03.542484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:48:43.734854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108995+0000","last_change":"2024-03-16T11:04:00.282168+0000","last_active":"2024-03-16T11:05:03.108995+0000","last_peered":"2024-03-16T11:05:03.108995+0000","last_clean":"2024-03-16T11:05:03.108995+0000","last_became_active":"2024-03-16T11:04:00.281753+0000","last_became_peered":"2024-03-16T11:04:00.281753+0000","last_unstale":"2024-03-16T11:05:03.108995+0000","last_undegraded":"2024-03-16T11:05:03.108995+0000","last_fullsized":"2024-03-16T11:05:03.108995+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:13.223955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.111462+0000","last_change":"2024-03-16T11:05:03.111462+0000","last_active":"2024-03-16T11:05:03.111462+0000","last_peered":"2024-03-16T11:05:03.111462+0000","last_clean":"2024-03-16T11:05:03.111462+0000","last_became_active":"2024-03-16T11:05:03.111087+0000","last_became_peered":"2024-03-16T11:05:03.111087+0000","last_unstale":"2024-03-16T11:05:03.111462+0000","last_undegraded":"2024-03-16T11:05:03.111462+0000","last_fullsized":"2024-03-16T11:05:03.111462+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:09.339732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105242+0000","last_change":"2024-03-16T11:04:00.292446+0000","last_active":"2024-03-16T11:05:03.105242+0000","last_peered":"2024-03-16T11:05:03.105242+0000","last_clean":"2024-03-16T11:05:03.105242+0000","last_became_active":"2024-03-16T11:04:00.292079+0000","last_became_peered":"2024-03-16T11:04:00.292079+0000","last_unstale":"2024-03-16T11:05:03.105242+0000","last_undegraded":"2024-03-16T11:05:03.105242+0000","last_fullsized":"2024-03-16T11:05:03.105242+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.325945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106536+0000","last_change":"2024-03-16T11:04:00.294644+0000","last_active":"2024-03-16T11:05:03.106536+0000","last_peered":"2024-03-16T11:05:03.106536+0000","last_clean":"2024-03-16T11:05:03.106536+0000","last_became_active":"2024-03-16T11:04:00.294283+0000","last_became_peered":"2024-03-16T11:04:00.294283+0000","last_unstale":"2024-03-16T11:05:03.106536+0000","last_undegraded":"2024-03-16T11:05:03.106536+0000","last_fullsized":"2024-03-16T11:05:03.106536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:43.125823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101215+0000","last_change":"2024-03-16T11:04:00.302512+0000","last_active":"2024-03-16T11:05:03.101215+0000","last_peered":"2024-03-16T11:05:03.101215+0000","last_clean":"2024-03-16T11:05:03.101215+0000","last_became_active":"2024-03-16T11:04:00.301891+0000","last_became_peered":"2024-03-16T11:04:00.301891+0000","last_unstale":"2024-03-16T11:05:03.101215+0000","last_undegraded":"2024-03-16T11:05:03.101215+0000","last_fullsized":"2024-03-16T11:05:03.101215+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:15.112295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106507+0000","last_change":"2024-03-16T11:04:00.293327+0000","last_active":"2024-03-16T11:05:03.106507+0000","last_peered":"2024-03-16T11:05:03.106507+0000","last_clean":"2024-03-16T11:05:03.106507+0000","last_became_active":"2024-03-16T11:04:00.292971+0000","last_became_peered":"2024-03-16T11:04:00.292971+0000","last_unstale":"2024-03-16T11:05:03.106507+0000","last_undegraded":"2024-03-16T11:05:03.106507+0000","last_fullsized":"2024-03-16T11:05:03.106507+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:48:48.565861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105615+0000","last_change":"2024-03-16T11:04:00.298956+0000","last_active":"2024-03-16T11:05:03.105615+0000","last_peered":"2024-03-16T11:05:03.105615+0000","last_clean":"2024-03-16T11:05:03.105615+0000","last_became_active":"2024-03-16T11:04:00.298508+0000","last_became_peered":"2024-03-16T11:04:00.298508+0000","last_unstale":"2024-03-16T11:05:03.105615+0000","last_undegraded":"2024-03-16T11:05:03.105615+0000","last_fullsized":"2024-03-16T11:05:03.105615+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:25:46.576514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542435+0000","last_change":"2024-03-16T11:04:00.294453+0000","last_active":"2024-03-16T11:05:03.542435+0000","last_peered":"2024-03-16T11:05:03.542435+0000","last_clean":"2024-03-16T11:05:03.542435+0000","last_became_active":"2024-03-16T11:04:00.294084+0000","last_became_peered":"2024-03-16T11:04:00.294084+0000","last_unstale":"2024-03-16T11:05:03.542435+0000","last_undegraded":"2024-03-16T11:05:03.542435+0000","last_fullsized":"2024-03-16T11:05:03.542435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:12.924835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103504+0000","last_change":"2024-03-16T11:04:00.293149+0000","last_active":"2024-03-16T11:05:03.103504+0000","last_peered":"2024-03-16T11:05:03.103504+0000","last_clean":"2024-03-16T11:05:03.103504+0000","last_became_active":"2024-03-16T11:04:00.292839+0000","last_became_peered":"2024-03-16T11:04:00.292839+0000","last_unstale":"2024-03-16T11:05:03.103504+0000","last_undegraded":"2024-03-16T11:05:03.103504+0000","last_fullsized":"2024-03-16T11:05:03.103504+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:34:38.446918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105081+0000","last_change":"2024-03-16T11:04:00.285748+0000","last_active":"2024-03-16T11:05:03.105081+0000","last_peered":"2024-03-16T11:05:03.105081+0000","last_clean":"2024-03-16T11:05:03.105081+0000","last_became_active":"2024-03-16T11:04:00.285394+0000","last_became_peered":"2024-03-16T11:04:00.285394+0000","last_unstale":"2024-03-16T11:05:03.105081+0000","last_undegraded":"2024-03-16T11:05:03.105081+0000","last_fullsized":"2024-03-16T11:05:03.105081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:02.113271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.104818+0000","last_change":"2024-03-16T11:04:00.301783+0000","last_active":"2024-03-16T11:05:03.104818+0000","last_peered":"2024-03-16T11:05:03.104818+0000","last_clean":"2024-03-16T11:05:03.104818+0000","last_became_active":"2024-03-16T11:04:00.301313+0000","last_became_peered":"2024-03-16T11:04:00.301313+0000","last_unstale":"2024-03-16T11:05:03.104818+0000","last_undegraded":"2024-03-16T11:05:03.104818+0000","last_fullsized":"2024-03-16T11:05:03.104818+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:19:32.243602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.100914+0000","last_change":"2024-03-16T11:04:00.301375+0000","last_active":"2024-03-16T11:05:03.100914+0000","last_peered":"2024-03-16T11:05:03.100914+0000","last_clean":"2024-03-16T11:05:03.100914+0000","last_became_active":"2024-03-16T11:04:00.301013+0000","last_became_peered":"2024-03-16T11:04:00.301013+0000","last_unstale":"2024-03-16T11:05:03.100914+0000","last_undegraded":"2024-03-16T11:05:03.100914+0000","last_fullsized":"2024-03-16T11:05:03.100914+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:21:52.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106418+0000","last_change":"2024-03-16T11:04:00.296287+0000","last_active":"2024-03-16T11:05:03.106418+0000","last_peered":"2024-03-16T11:05:03.106418+0000","last_clean":"2024-03-16T11:05:03.106418+0000","last_became_active":"2024-03-16T11:04:00.295972+0000","last_became_peered":"2024-03-16T11:04:00.295972+0000","last_unstale":"2024-03-16T11:05:03.106418+0000","last_undegraded":"2024-03-16T11:05:03.106418+0000","last_fullsized":"2024-03-16T11:05:03.106418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:14.586052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102850+0000","last_change":"2024-03-16T11:04:00.295551+0000","last_active":"2024-03-16T11:05:03.102850+0000","last_peered":"2024-03-16T11:05:03.102850+0000","last_clean":"2024-03-16T11:05:03.102850+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:03.102850+0000","last_undegraded":"2024-03-16T11:05:03.102850+0000","last_fullsized":"2024-03-16T11:05:03.102850+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:53.495224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.549105+0000","last_change":"2024-03-16T11:04:00.290455+0000","last_active":"2024-03-16T11:05:03.549105+0000","last_peered":"2024-03-16T11:05:03.549105+0000","last_clean":"2024-03-16T11:05:03.549105+0000","last_became_active":"2024-03-16T11:04:00.290057+0000","last_became_peered":"2024-03-16T11:04:00.290057+0000","last_unstale":"2024-03-16T11:05:03.549105+0000","last_undegraded":"2024-03-16T11:05:03.549105+0000","last_fullsized":"2024-03-16T11:05:03.549105+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:57:43.677050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102676+0000","last_change":"2024-03-16T11:03:59.285059+0000","last_active":"2024-03-16T11:05:03.102676+0000","last_peered":"2024-03-16T11:05:03.102676+0000","last_clean":"2024-03-16T11:05:03.102676+0000","last_became_active":"2024-03-16T11:03:59.284745+0000","last_became_peered":"2024-03-16T11:03:59.284745+0000","last_unstale":"2024-03-16T11:05:03.102676+0000","last_undegraded":"2024-03-16T11:05:03.102676+0000","last_fullsized":"2024-03-16T11:05:03.102676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:12.516958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107404+0000","last_change":"2024-03-16T11:04:00.292933+0000","last_active":"2024-03-16T11:05:03.107404+0000","last_peered":"2024-03-16T11:05:03.107404+0000","last_clean":"2024-03-16T11:05:03.107404+0000","last_became_active":"2024-03-16T11:04:00.292637+0000","last_became_peered":"2024-03-16T11:04:00.292637+0000","last_unstale":"2024-03-16T11:05:03.107404+0000","last_undegraded":"2024-03-16T11:05:03.107404+0000","last_fullsized":"2024-03-16T11:05:03.107404+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:22.231735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107896+0000","last_change":"2024-03-16T11:03:59.285267+0000","last_active":"2024-03-16T11:05:03.107896+0000","last_peered":"2024-03-16T11:05:03.107896+0000","last_clean":"2024-03-16T11:05:03.107896+0000","last_became_active":"2024-03-16T11:03:59.284890+0000","last_became_peered":"2024-03-16T11:03:59.284890+0000","last_unstale":"2024-03-16T11:05:03.107896+0000","last_undegraded":"2024-03-16T11:05:03.107896+0000","last_fullsized":"2024-03-16T11:05:03.107896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:53.625612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105722+0000","last_change":"2024-03-16T11:04:00.292660+0000","last_active":"2024-03-16T11:05:03.105722+0000","last_peered":"2024-03-16T11:05:03.105722+0000","last_clean":"2024-03-16T11:05:03.105722+0000","last_became_active":"2024-03-16T11:04:00.292366+0000","last_became_peered":"2024-03-16T11:04:00.292366+0000","last_unstale":"2024-03-16T11:05:03.105722+0000","last_undegraded":"2024-03-16T11:05:03.105722+0000","last_fullsized":"2024-03-16T11:05:03.105722+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:36:44.331456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109296+0000","last_change":"2024-03-16T11:03:59.286370+0000","last_active":"2024-03-16T11:05:03.109296+0000","last_peered":"2024-03-16T11:05:03.109296+0000","last_clean":"2024-03-16T11:05:03.109296+0000","last_became_active":"2024-03-16T11:03:59.286065+0000","last_became_peered":"2024-03-16T11:03:59.286065+0000","last_unstale":"2024-03-16T11:05:03.109296+0000","last_undegraded":"2024-03-16T11:05:03.109296+0000","last_fullsized":"2024-03-16T11:05:03.109296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:20.126848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101112+0000","last_change":"2024-03-16T11:04:00.290864+0000","last_active":"2024-03-16T11:05:03.101112+0000","last_peered":"2024-03-16T11:05:03.101112+0000","last_clean":"2024-03-16T11:05:03.101112+0000","last_became_active":"2024-03-16T11:04:00.290537+0000","last_became_peered":"2024-03-16T11:04:00.290537+0000","last_unstale":"2024-03-16T11:05:03.101112+0000","last_undegraded":"2024-03-16T11:05:03.101112+0000","last_fullsized":"2024-03-16T11:05:03.101112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:52.991767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109155+0000","last_change":"2024-03-16T11:03:59.281663+0000","last_active":"2024-03-16T11:05:03.109155+0000","last_peered":"2024-03-16T11:05:03.109155+0000","last_clean":"2024-03-16T11:05:03.109155+0000","last_became_active":"2024-03-16T11:03:59.281328+0000","last_became_peered":"2024-03-16T11:03:59.281328+0000","last_unstale":"2024-03-16T11:05:03.109155+0000","last_undegraded":"2024-03-16T11:05:03.109155+0000","last_fullsized":"2024-03-16T11:05:03.109155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:44.511109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106586+0000","last_change":"2024-03-16T11:05:03.106586+0000","last_active":"2024-03-16T11:05:03.106586+0000","last_peered":"2024-03-16T11:05:03.106586+0000","last_clean":"2024-03-16T11:05:03.106586+0000","last_became_active":"2024-03-16T11:05:03.106233+0000","last_became_peered":"2024-03-16T11:05:03.106233+0000","last_unstale":"2024-03-16T11:05:03.106586+0000","last_undegraded":"2024-03-16T11:05:03.106586+0000","last_fullsized":"2024-03-16T11:05:03.106586+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:19:30.480318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":98,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:13.545520+0000","last_change":"2024-03-16T11:03:59.286300+0000","last_active":"2024-03-16T11:05:13.545520+0000","last_peered":"2024-03-16T11:05:13.545520+0000","last_clean":"2024-03-16T11:05:13.545520+0000","last_became_active":"2024-03-16T11:03:59.285764+0000","last_became_peered":"2024-03-16T11:03:59.285764+0000","last_unstale":"2024-03-16T11:05:13.545520+0000","last_undegraded":"2024-03-16T11:05:13.545520+0000","last_fullsized":"2024-03-16T11:05:13.545520+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:49.084245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108999+0000","last_change":"2024-03-16T11:04:00.289152+0000","last_active":"2024-03-16T11:05:03.108999+0000","last_peered":"2024-03-16T11:05:03.108999+0000","last_clean":"2024-03-16T11:05:03.108999+0000","last_became_active":"2024-03-16T11:04:00.288837+0000","last_became_peered":"2024-03-16T11:04:00.288837+0000","last_unstale":"2024-03-16T11:05:03.108999+0000","last_undegraded":"2024-03-16T11:05:03.108999+0000","last_fullsized":"2024-03-16T11:05:03.108999+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:45:01.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106073+0000","last_change":"2024-03-16T11:03:59.281505+0000","last_active":"2024-03-16T11:05:03.106073+0000","last_peered":"2024-03-16T11:05:03.106073+0000","last_clean":"2024-03-16T11:05:03.106073+0000","last_became_active":"2024-03-16T11:03:59.281175+0000","last_became_peered":"2024-03-16T11:03:59.281175+0000","last_unstale":"2024-03-16T11:05:03.106073+0000","last_undegraded":"2024-03-16T11:05:03.106073+0000","last_fullsized":"2024-03-16T11:05:03.106073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:12.463914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103210+0000","last_change":"2024-03-16T11:04:00.297769+0000","last_active":"2024-03-16T11:05:03.103210+0000","last_peered":"2024-03-16T11:05:03.103210+0000","last_clean":"2024-03-16T11:05:03.103210+0000","last_became_active":"2024-03-16T11:04:00.297061+0000","last_became_peered":"2024-03-16T11:04:00.297061+0000","last_unstale":"2024-03-16T11:05:03.103210+0000","last_undegraded":"2024-03-16T11:05:03.103210+0000","last_fullsized":"2024-03-16T11:05:03.103210+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:42.594315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103214+0000","last_change":"2024-03-16T11:03:59.290260+0000","last_active":"2024-03-16T11:05:03.103214+0000","last_peered":"2024-03-16T11:05:03.103214+0000","last_clean":"2024-03-16T11:05:03.103214+0000","last_became_active":"2024-03-16T11:03:59.289843+0000","last_became_peered":"2024-03-16T11:03:59.289843+0000","last_unstale":"2024-03-16T11:05:03.103214+0000","last_undegraded":"2024-03-16T11:05:03.103214+0000","last_fullsized":"2024-03-16T11:05:03.103214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:22.483488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106580+0000","last_change":"2024-03-16T11:04:00.287873+0000","last_active":"2024-03-16T11:05:03.106580+0000","last_peered":"2024-03-16T11:05:03.106580+0000","last_clean":"2024-03-16T11:05:03.106580+0000","last_became_active":"2024-03-16T11:04:00.287002+0000","last_became_peered":"2024-03-16T11:04:00.287002+0000","last_unstale":"2024-03-16T11:05:03.106580+0000","last_undegraded":"2024-03-16T11:05:03.106580+0000","last_fullsized":"2024-03-16T11:05:03.106580+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:57.851726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102815+0000","last_change":"2024-03-16T11:03:59.284745+0000","last_active":"2024-03-16T11:05:03.102815+0000","last_peered":"2024-03-16T11:05:03.102815+0000","last_clean":"2024-03-16T11:05:03.102815+0000","last_became_active":"2024-03-16T11:03:59.284325+0000","last_became_peered":"2024-03-16T11:03:59.284325+0000","last_unstale":"2024-03-16T11:05:03.102815+0000","last_undegraded":"2024-03-16T11:05:03.102815+0000","last_fullsized":"2024-03-16T11:05:03.102815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.537787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102951+0000","last_change":"2024-03-16T11:04:00.290698+0000","last_active":"2024-03-16T11:05:03.102951+0000","last_peered":"2024-03-16T11:05:03.102951+0000","last_clean":"2024-03-16T11:05:03.102951+0000","last_became_active":"2024-03-16T11:04:00.290348+0000","last_became_peered":"2024-03-16T11:04:00.290348+0000","last_unstale":"2024-03-16T11:05:03.102951+0000","last_undegraded":"2024-03-16T11:05:03.102951+0000","last_fullsized":"2024-03-16T11:05:03.102951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:44.114305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:13.544965+0000","last_change":"2024-03-16T11:03:59.280705+0000","last_active":"2024-03-16T11:05:13.544965+0000","last_peered":"2024-03-16T11:05:13.544965+0000","last_clean":"2024-03-16T11:05:13.544965+0000","last_became_active":"2024-03-16T11:03:59.280372+0000","last_became_peered":"2024-03-16T11:03:59.280372+0000","last_unstale":"2024-03-16T11:05:13.544965+0000","last_undegraded":"2024-03-16T11:05:13.544965+0000","last_fullsized":"2024-03-16T11:05:13.544965+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:54.187322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101124+0000","last_change":"2024-03-16T11:04:00.293867+0000","last_active":"2024-03-16T11:05:03.101124+0000","last_peered":"2024-03-16T11:05:03.101124+0000","last_clean":"2024-03-16T11:05:03.101124+0000","last_became_active":"2024-03-16T11:04:00.293330+0000","last_became_peered":"2024-03-16T11:04:00.293330+0000","last_unstale":"2024-03-16T11:05:03.101124+0000","last_undegraded":"2024-03-16T11:05:03.101124+0000","last_fullsized":"2024-03-16T11:05:03.101124+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:54.798346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105547+0000","last_change":"2024-03-16T11:03:59.284202+0000","last_active":"2024-03-16T11:05:03.105547+0000","last_peered":"2024-03-16T11:05:03.105547+0000","last_clean":"2024-03-16T11:05:03.105547+0000","last_became_active":"2024-03-16T11:03:59.283656+0000","last_became_peered":"2024-03-16T11:03:59.283656+0000","last_unstale":"2024-03-16T11:05:03.105547+0000","last_undegraded":"2024-03-16T11:05:03.105547+0000","last_fullsized":"2024-03-16T11:05:03.105547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:44.760307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.111443+0000","last_change":"2024-03-16T11:05:03.111443+0000","last_active":"2024-03-16T11:05:03.111443+0000","last_peered":"2024-03-16T11:05:03.111443+0000","last_clean":"2024-03-16T11:05:03.111443+0000","last_became_active":"2024-03-16T11:05:03.111027+0000","last_became_peered":"2024-03-16T11:05:03.111027+0000","last_unstale":"2024-03-16T11:05:03.111443+0000","last_undegraded":"2024-03-16T11:05:03.111443+0000","last_fullsized":"2024-03-16T11:05:03.111443+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:31.844283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103330+0000","last_change":"2024-03-16T11:03:59.288268+0000","last_active":"2024-03-16T11:05:03.103330+0000","last_peered":"2024-03-16T11:05:03.103330+0000","last_clean":"2024-03-16T11:05:03.103330+0000","last_became_active":"2024-03-16T11:03:59.287861+0000","last_became_peered":"2024-03-16T11:03:59.287861+0000","last_unstale":"2024-03-16T11:05:03.103330+0000","last_undegraded":"2024-03-16T11:05:03.103330+0000","last_fullsized":"2024-03-16T11:05:03.103330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:34:34.248408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101261+0000","last_change":"2024-03-16T11:04:00.306490+0000","last_active":"2024-03-16T11:05:03.101261+0000","last_peered":"2024-03-16T11:05:03.101261+0000","last_clean":"2024-03-16T11:05:03.101261+0000","last_became_active":"2024-03-16T11:04:00.305874+0000","last_became_peered":"2024-03-16T11:04:00.305874+0000","last_unstale":"2024-03-16T11:05:03.101261+0000","last_undegraded":"2024-03-16T11:05:03.101261+0000","last_fullsized":"2024-03-16T11:05:03.101261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'26","reported_seq":196,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:14.546110+0000","last_change":"2024-03-16T11:03:59.280714+0000","last_active":"2024-03-16T11:05:14.546110+0000","last_peered":"2024-03-16T11:05:14.546110+0000","last_clean":"2024-03-16T11:05:14.546110+0000","last_became_active":"2024-03-16T11:03:59.280398+0000","last_became_peered":"2024-03-16T11:03:59.280398+0000","last_unstale":"2024-03-16T11:05:14.546110+0000","last_undegraded":"2024-03-16T11:05:14.546110+0000","last_fullsized":"2024-03-16T11:05:14.546110+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:34.256125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":132,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109217+0000","last_change":"2024-03-16T11:04:00.285669+0000","last_active":"2024-03-16T11:05:03.109217+0000","last_peered":"2024-03-16T11:05:03.109217+0000","last_clean":"2024-03-16T11:05:03.109217+0000","last_became_active":"2024-03-16T11:04:00.285337+0000","last_became_peered":"2024-03-16T11:04:00.285337+0000","last_unstale":"2024-03-16T11:05:03.109217+0000","last_undegraded":"2024-03-16T11:05:03.109217+0000","last_fullsized":"2024-03-16T11:05:03.109217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:36.925350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548506+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548506+0000","last_peered":"2024-03-16T11:05:03.548506+0000","last_clean":"2024-03-16T11:05:03.548506+0000","last_became_active":"2024-03-16T11:03:59.289357+0000","last_became_peered":"2024-03-16T11:03:59.289357+0000","last_unstale":"2024-03-16T11:05:03.548506+0000","last_undegraded":"2024-03-16T11:05:03.548506+0000","last_fullsized":"2024-03-16T11:05:03.548506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:26:35.860091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103313+0000","last_change":"2024-03-16T11:04:00.298488+0000","last_active":"2024-03-16T11:05:03.103313+0000","last_peered":"2024-03-16T11:05:03.103313+0000","last_clean":"2024-03-16T11:05:03.103313+0000","last_became_active":"2024-03-16T11:04:00.298202+0000","last_became_peered":"2024-03-16T11:04:00.298202+0000","last_unstale":"2024-03-16T11:05:03.103313+0000","last_undegraded":"2024-03-16T11:05:03.103313+0000","last_fullsized":"2024-03-16T11:05:03.103313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:48.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102961+0000","last_change":"2024-03-16T11:03:59.287066+0000","last_active":"2024-03-16T11:05:03.102961+0000","last_peered":"2024-03-16T11:05:03.102961+0000","last_clean":"2024-03-16T11:05:03.102961+0000","last_became_active":"2024-03-16T11:03:59.286731+0000","last_became_peered":"2024-03-16T11:03:59.286731+0000","last_unstale":"2024-03-16T11:05:03.102961+0000","last_undegraded":"2024-03-16T11:05:03.102961+0000","last_fullsized":"2024-03-16T11:05:03.102961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:51.443833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105488+0000","last_change":"2024-03-16T11:04:00.294450+0000","last_active":"2024-03-16T11:05:03.105488+0000","last_peered":"2024-03-16T11:05:03.105488+0000","last_clean":"2024-03-16T11:05:03.105488+0000","last_became_active":"2024-03-16T11:04:00.293769+0000","last_became_peered":"2024-03-16T11:04:00.293769+0000","last_unstale":"2024-03-16T11:05:03.105488+0000","last_undegraded":"2024-03-16T11:05:03.105488+0000","last_fullsized":"2024-03-16T11:05:03.105488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:02.755907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":97,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:12.552251+0000","last_change":"2024-03-16T11:03:59.286197+0000","last_active":"2024-03-16T11:05:12.552251+0000","last_peered":"2024-03-16T11:05:12.552251+0000","last_clean":"2024-03-16T11:05:12.552251+0000","last_became_active":"2024-03-16T11:03:59.285757+0000","last_became_peered":"2024-03-16T11:03:59.285757+0000","last_unstale":"2024-03-16T11:05:12.552251+0000","last_undegraded":"2024-03-16T11:05:12.552251+0000","last_fullsized":"2024-03-16T11:05:12.552251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:47.868106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105321+0000","last_change":"2024-03-16T11:04:00.296709+0000","last_active":"2024-03-16T11:05:03.105321+0000","last_peered":"2024-03-16T11:05:03.105321+0000","last_clean":"2024-03-16T11:05:03.105321+0000","last_became_active":"2024-03-16T11:04:00.296372+0000","last_became_peered":"2024-03-16T11:04:00.296372+0000","last_unstale":"2024-03-16T11:05:03.105321+0000","last_undegraded":"2024-03-16T11:05:03.105321+0000","last_fullsized":"2024-03-16T11:05:03.105321+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:37.948866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108997+0000","last_change":"2024-03-16T11:03:59.283831+0000","last_active":"2024-03-16T11:05:03.108997+0000","last_peered":"2024-03-16T11:05:03.108997+0000","last_clean":"2024-03-16T11:05:03.108997+0000","last_became_active":"2024-03-16T11:03:59.283404+0000","last_became_peered":"2024-03-16T11:03:59.283404+0000","last_unstale":"2024-03-16T11:05:03.108997+0000","last_undegraded":"2024-03-16T11:05:03.108997+0000","last_fullsized":"2024-03-16T11:05:03.108997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:26:34.012597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102890+0000","last_change":"2024-03-16T11:04:00.299703+0000","last_active":"2024-03-16T11:05:03.102890+0000","last_peered":"2024-03-16T11:05:03.102890+0000","last_clean":"2024-03-16T11:05:03.102890+0000","last_became_active":"2024-03-16T11:04:00.299321+0000","last_became_peered":"2024-03-16T11:04:00.299321+0000","last_unstale":"2024-03-16T11:05:03.102890+0000","last_undegraded":"2024-03-16T11:05:03.102890+0000","last_fullsized":"2024-03-16T11:05:03.102890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.529281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109510+0000","last_change":"2024-03-16T11:03:59.277837+0000","last_active":"2024-03-16T11:05:03.109510+0000","last_peered":"2024-03-16T11:05:03.109510+0000","last_clean":"2024-03-16T11:05:03.109510+0000","last_became_active":"2024-03-16T11:03:59.277515+0000","last_became_peered":"2024-03-16T11:03:59.277515+0000","last_unstale":"2024-03-16T11:05:03.109510+0000","last_undegraded":"2024-03-16T11:05:03.109510+0000","last_fullsized":"2024-03-16T11:05:03.109510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:05.116990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106199+0000","last_change":"2024-03-16T11:04:00.292998+0000","last_active":"2024-03-16T11:05:03.106199+0000","last_peered":"2024-03-16T11:05:03.106199+0000","last_clean":"2024-03-16T11:05:03.106199+0000","last_became_active":"2024-03-16T11:04:00.292608+0000","last_became_peered":"2024-03-16T11:04:00.292608+0000","last_unstale":"2024-03-16T11:05:03.106199+0000","last_undegraded":"2024-03-16T11:05:03.106199+0000","last_fullsized":"2024-03-16T11:05:03.106199+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:41.996826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108505+0000","last_change":"2024-03-16T11:05:03.108505+0000","last_active":"2024-03-16T11:05:03.108505+0000","last_peered":"2024-03-16T11:05:03.108505+0000","last_clean":"2024-03-16T11:05:03.108505+0000","last_became_active":"2024-03-16T11:05:03.108132+0000","last_became_peered":"2024-03-16T11:05:03.108132+0000","last_unstale":"2024-03-16T11:05:03.108505+0000","last_undegraded":"2024-03-16T11:05:03.108505+0000","last_fullsized":"2024-03-16T11:05:03.108505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:50.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106434+0000","last_change":"2024-03-16T11:04:00.288897+0000","last_active":"2024-03-16T11:05:03.106434+0000","last_peered":"2024-03-16T11:05:03.106434+0000","last_clean":"2024-03-16T11:05:03.106434+0000","last_became_active":"2024-03-16T11:04:00.288493+0000","last_became_peered":"2024-03-16T11:04:00.288493+0000","last_unstale":"2024-03-16T11:05:03.106434+0000","last_undegraded":"2024-03-16T11:05:03.106434+0000","last_fullsized":"2024-03-16T11:05:03.106434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:31.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105515+0000","last_change":"2024-03-16T11:03:59.291595+0000","last_active":"2024-03-16T11:05:03.105515+0000","last_peered":"2024-03-16T11:05:03.105515+0000","last_clean":"2024-03-16T11:05:03.105515+0000","last_became_active":"2024-03-16T11:03:59.291159+0000","last_became_peered":"2024-03-16T11:03:59.291159+0000","last_unstale":"2024-03-16T11:05:03.105515+0000","last_undegraded":"2024-03-16T11:05:03.105515+0000","last_fullsized":"2024-03-16T11:05:03.105515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:03.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108818+0000","last_change":"2024-03-16T11:05:03.108818+0000","last_active":"2024-03-16T11:05:03.108818+0000","last_peered":"2024-03-16T11:05:03.108818+0000","last_clean":"2024-03-16T11:05:03.108818+0000","last_became_active":"2024-03-16T11:05:03.108479+0000","last_became_peered":"2024-03-16T11:05:03.108479+0000","last_unstale":"2024-03-16T11:05:03.108818+0000","last_undegraded":"2024-03-16T11:05:03.108818+0000","last_fullsized":"2024-03-16T11:05:03.108818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:10:11.329743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103022+0000","last_change":"2024-03-16T11:04:00.298400+0000","last_active":"2024-03-16T11:05:03.103022+0000","last_peered":"2024-03-16T11:05:03.103022+0000","last_clean":"2024-03-16T11:05:03.103022+0000","last_became_active":"2024-03-16T11:04:00.298026+0000","last_became_peered":"2024-03-16T11:04:00.298026+0000","last_unstale":"2024-03-16T11:05:03.103022+0000","last_undegraded":"2024-03-16T11:05:03.103022+0000","last_fullsized":"2024-03-16T11:05:03.103022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:08.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:12.551778+0000","last_change":"2024-03-16T11:03:59.287528+0000","last_active":"2024-03-16T11:05:12.551778+0000","last_peered":"2024-03-16T11:05:12.551778+0000","last_clean":"2024-03-16T11:05:12.551778+0000","last_became_active":"2024-03-16T11:03:59.287218+0000","last_became_peered":"2024-03-16T11:03:59.287218+0000","last_unstale":"2024-03-16T11:05:12.551778+0000","last_undegraded":"2024-03-16T11:05:12.551778+0000","last_fullsized":"2024-03-16T11:05:12.551778+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:19.803233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106614+0000","last_change":"2024-03-16T11:05:03.106099+0000","last_active":"2024-03-16T11:05:03.106614+0000","last_peered":"2024-03-16T11:05:03.106614+0000","last_clean":"2024-03-16T11:05:03.106614+0000","last_became_active":"2024-03-16T11:05:03.105412+0000","last_became_peered":"2024-03-16T11:05:03.105412+0000","last_unstale":"2024-03-16T11:05:03.106614+0000","last_undegraded":"2024-03-16T11:05:03.106614+0000","last_fullsized":"2024-03-16T11:05:03.106614+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:29:33.213676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.541779+0000","last_change":"2024-03-16T11:03:59.281251+0000","last_active":"2024-03-16T11:05:03.541779+0000","last_peered":"2024-03-16T11:05:03.541779+0000","last_clean":"2024-03-16T11:05:03.541779+0000","last_became_active":"2024-03-16T11:03:59.280830+0000","last_became_peered":"2024-03-16T11:03:59.280830+0000","last_unstale":"2024-03-16T11:05:03.541779+0000","last_undegraded":"2024-03-16T11:05:03.541779+0000","last_fullsized":"2024-03-16T11:05:03.541779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:03:04.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105426+0000","last_change":"2024-03-16T11:02:48.497749+0000","last_active":"2024-03-16T11:05:03.105426+0000","last_peered":"2024-03-16T11:05:03.105426+0000","last_clean":"2024-03-16T11:05:03.105426+0000","last_became_active":"2024-03-16T11:02:48.497157+0000","last_became_peered":"2024-03-16T11:02:48.497157+0000","last_unstale":"2024-03-16T11:05:03.105426+0000","last_undegraded":"2024-03-16T11:05:03.105426+0000","last_fullsized":"2024-03-16T11:05:03.105426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_clean_scrub_stamp":"2024-03-16T10:59:26.309639+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:04.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102952+0000","last_change":"2024-03-16T11:04:00.300821+0000","last_active":"2024-03-16T11:05:03.102952+0000","last_peered":"2024-03-16T11:05:03.102952+0000","last_clean":"2024-03-16T11:05:03.102952+0000","last_became_active":"2024-03-16T11:04:00.300510+0000","last_became_peered":"2024-03-16T11:04:00.300510+0000","last_unstale":"2024-03-16T11:05:03.102952+0000","last_undegraded":"2024-03-16T11:05:03.102952+0000","last_fullsized":"2024-03-16T11:05:03.102952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:57.035167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105571+0000","last_change":"2024-03-16T11:03:59.283609+0000","last_active":"2024-03-16T11:05:03.105571+0000","last_peered":"2024-03-16T11:05:03.105571+0000","last_clean":"2024-03-16T11:05:03.105571+0000","last_became_active":"2024-03-16T11:03:59.283227+0000","last_became_peered":"2024-03-16T11:03:59.283227+0000","last_unstale":"2024-03-16T11:05:03.105571+0000","last_undegraded":"2024-03-16T11:05:03.105571+0000","last_fullsized":"2024-03-16T11:05:03.105571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:18.904202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.112457+0000","last_change":"2024-03-16T11:05:03.112071+0000","last_active":"2024-03-16T11:05:03.112457+0000","last_peered":"2024-03-16T11:05:03.112457+0000","last_clean":"2024-03-16T11:05:03.112457+0000","last_became_active":"2024-03-16T11:05:03.108434+0000","last_became_peered":"2024-03-16T11:05:03.108434+0000","last_unstale":"2024-03-16T11:05:03.112457+0000","last_undegraded":"2024-03-16T11:05:03.112457+0000","last_fullsized":"2024-03-16T11:05:03.112457+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:24.758886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103481+0000","last_change":"2024-03-16T11:04:00.285134+0000","last_active":"2024-03-16T11:05:03.103481+0000","last_peered":"2024-03-16T11:05:03.103481+0000","last_clean":"2024-03-16T11:05:03.103481+0000","last_became_active":"2024-03-16T11:04:00.284705+0000","last_became_peered":"2024-03-16T11:04:00.284705+0000","last_unstale":"2024-03-16T11:05:03.103481+0000","last_undegraded":"2024-03-16T11:05:03.103481+0000","last_fullsized":"2024-03-16T11:05:03.103481+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:38.637970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102736+0000","last_change":"2024-03-16T11:03:59.289999+0000","last_active":"2024-03-16T11:05:03.102736+0000","last_peered":"2024-03-16T11:05:03.102736+0000","last_clean":"2024-03-16T11:05:03.102736+0000","last_became_active":"2024-03-16T11:03:59.289694+0000","last_became_peered":"2024-03-16T11:03:59.289694+0000","last_unstale":"2024-03-16T11:05:03.102736+0000","last_undegraded":"2024-03-16T11:05:03.102736+0000","last_fullsized":"2024-03-16T11:05:03.102736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:16.037387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107698+0000","last_change":"2024-03-16T11:04:00.298148+0000","last_active":"2024-03-16T11:05:03.107698+0000","last_peered":"2024-03-16T11:05:03.107698+0000","last_clean":"2024-03-16T11:05:03.107698+0000","last_became_active":"2024-03-16T11:04:00.297822+0000","last_became_peered":"2024-03-16T11:04:00.297822+0000","last_unstale":"2024-03-16T11:05:03.107698+0000","last_undegraded":"2024-03-16T11:05:03.107698+0000","last_fullsized":"2024-03-16T11:05:03.107698+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:09:28.645317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542005+0000","last_change":"2024-03-16T11:03:59.292207+0000","last_active":"2024-03-16T11:05:03.542005+0000","last_peered":"2024-03-16T11:05:03.542005+0000","last_clean":"2024-03-16T11:05:03.542005+0000","last_became_active":"2024-03-16T11:03:59.291859+0000","last_became_peered":"2024-03-16T11:03:59.291859+0000","last_unstale":"2024-03-16T11:05:03.542005+0000","last_undegraded":"2024-03-16T11:05:03.542005+0000","last_fullsized":"2024-03-16T11:05:03.542005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:55.365720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105643+0000","last_change":"2024-03-16T11:04:00.291261+0000","last_active":"2024-03-16T11:05:03.105643+0000","last_peered":"2024-03-16T11:05:03.105643+0000","last_clean":"2024-03-16T11:05:03.105643+0000","last_became_active":"2024-03-16T11:04:00.290832+0000","last_became_peered":"2024-03-16T11:04:00.290832+0000","last_unstale":"2024-03-16T11:05:03.105643+0000","last_undegraded":"2024-03-16T11:05:03.105643+0000","last_fullsized":"2024-03-16T11:05:03.105643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:10.418808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.100986+0000","last_change":"2024-03-16T11:03:59.288014+0000","last_active":"2024-03-16T11:05:03.100986+0000","last_peered":"2024-03-16T11:05:03.100986+0000","last_clean":"2024-03-16T11:05:03.100986+0000","last_became_active":"2024-03-16T11:03:59.287679+0000","last_became_peered":"2024-03-16T11:03:59.287679+0000","last_unstale":"2024-03-16T11:05:03.100986+0000","last_undegraded":"2024-03-16T11:05:03.100986+0000","last_fullsized":"2024-03-16T11:05:03.100986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:16:39.156224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108907+0000","last_change":"2024-03-16T11:04:00.288047+0000","last_active":"2024-03-16T11:05:03.108907+0000","last_peered":"2024-03-16T11:05:03.108907+0000","last_clean":"2024-03-16T11:05:03.108907+0000","last_became_active":"2024-03-16T11:04:00.287619+0000","last_became_peered":"2024-03-16T11:04:00.287619+0000","last_unstale":"2024-03-16T11:05:03.108907+0000","last_undegraded":"2024-03-16T11:05:03.108907+0000","last_fullsized":"2024-03-16T11:05:03.108907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:43.608349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106759+0000","last_change":"2024-03-16T11:03:59.288439+0000","last_active":"2024-03-16T11:05:03.106759+0000","last_peered":"2024-03-16T11:05:03.106759+0000","last_clean":"2024-03-16T11:05:03.106759+0000","last_became_active":"2024-03-16T11:03:59.288122+0000","last_became_peered":"2024-03-16T11:03:59.288122+0000","last_unstale":"2024-03-16T11:05:03.106759+0000","last_undegraded":"2024-03-16T11:05:03.106759+0000","last_fullsized":"2024-03-16T11:05:03.106759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:50:25.499608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105759+0000","last_change":"2024-03-16T11:04:00.290522+0000","last_active":"2024-03-16T11:05:03.105759+0000","last_peered":"2024-03-16T11:05:03.105759+0000","last_clean":"2024-03-16T11:05:03.105759+0000","last_became_active":"2024-03-16T11:04:00.290175+0000","last_became_peered":"2024-03-16T11:04:00.290175+0000","last_unstale":"2024-03-16T11:05:03.105759+0000","last_undegraded":"2024-03-16T11:05:03.105759+0000","last_fullsized":"2024-03-16T11:05:03.105759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:34.113278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548452+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548452+0000","last_peered":"2024-03-16T11:05:03.548452+0000","last_clean":"2024-03-16T11:05:03.548452+0000","last_became_active":"2024-03-16T11:03:59.289367+0000","last_became_peered":"2024-03-16T11:03:59.289367+0000","last_unstale":"2024-03-16T11:05:03.548452+0000","last_undegraded":"2024-03-16T11:05:03.548452+0000","last_fullsized":"2024-03-16T11:05:03.548452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:42:48.627116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107105+0000","last_change":"2024-03-16T11:04:00.287685+0000","last_active":"2024-03-16T11:05:03.107105+0000","last_peered":"2024-03-16T11:05:03.107105+0000","last_clean":"2024-03-16T11:05:03.107105+0000","last_became_active":"2024-03-16T11:04:00.287367+0000","last_became_peered":"2024-03-16T11:04:00.287367+0000","last_unstale":"2024-03-16T11:05:03.107105+0000","last_undegraded":"2024-03-16T11:05:03.107105+0000","last_fullsized":"2024-03-16T11:05:03.107105+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:25:52.741651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108923+0000","last_change":"2024-03-16T11:03:59.286293+0000","last_active":"2024-03-16T11:05:03.108923+0000","last_peered":"2024-03-16T11:05:03.108923+0000","last_clean":"2024-03-16T11:05:03.108923+0000","last_became_active":"2024-03-16T11:03:59.285817+0000","last_became_peered":"2024-03-16T11:03:59.285817+0000","last_unstale":"2024-03-16T11:05:03.108923+0000","last_undegraded":"2024-03-16T11:05:03.108923+0000","last_fullsized":"2024-03-16T11:05:03.108923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:38:12.093762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105889+0000","last_change":"2024-03-16T11:04:00.299391+0000","last_active":"2024-03-16T11:05:03.105889+0000","last_peered":"2024-03-16T11:05:03.105889+0000","last_clean":"2024-03-16T11:05:03.105889+0000","last_became_active":"2024-03-16T11:04:00.298931+0000","last_became_peered":"2024-03-16T11:04:00.298931+0000","last_unstale":"2024-03-16T11:05:03.105889+0000","last_undegraded":"2024-03-16T11:05:03.105889+0000","last_fullsized":"2024-03-16T11:05:03.105889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:12:11.900154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548534+0000","last_change":"2024-03-16T11:03:59.292319+0000","last_active":"2024-03-16T11:05:03.548534+0000","last_peered":"2024-03-16T11:05:03.548534+0000","last_clean":"2024-03-16T11:05:03.548534+0000","last_became_active":"2024-03-16T11:03:59.291953+0000","last_became_peered":"2024-03-16T11:03:59.291953+0000","last_unstale":"2024-03-16T11:05:03.548534+0000","last_undegraded":"2024-03-16T11:05:03.548534+0000","last_fullsized":"2024-03-16T11:05:03.548534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:41:04.615726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105749+0000","last_change":"2024-03-16T11:04:00.296544+0000","last_active":"2024-03-16T11:05:03.105749+0000","last_peered":"2024-03-16T11:05:03.105749+0000","last_clean":"2024-03-16T11:05:03.105749+0000","last_became_active":"2024-03-16T11:04:00.296078+0000","last_became_peered":"2024-03-16T11:04:00.296078+0000","last_unstale":"2024-03-16T11:05:03.105749+0000","last_undegraded":"2024-03-16T11:05:03.105749+0000","last_fullsized":"2024-03-16T11:05:03.105749+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:56:22.846734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103250+0000","last_change":"2024-03-16T11:03:59.287172+0000","last_active":"2024-03-16T11:05:03.103250+0000","last_peered":"2024-03-16T11:05:03.103250+0000","last_clean":"2024-03-16T11:05:03.103250+0000","last_became_active":"2024-03-16T11:03:59.286785+0000","last_became_peered":"2024-03-16T11:03:59.286785+0000","last_unstale":"2024-03-16T11:05:03.103250+0000","last_undegraded":"2024-03-16T11:05:03.103250+0000","last_fullsized":"2024-03-16T11:05:03.103250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:17:55.584618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.111534+0000","last_change":"2024-03-16T11:05:03.111534+0000","last_active":"2024-03-16T11:05:03.111534+0000","last_peered":"2024-03-16T11:05:03.111534+0000","last_clean":"2024-03-16T11:05:03.111534+0000","last_became_active":"2024-03-16T11:05:03.107517+0000","last_became_peered":"2024-03-16T11:05:03.107517+0000","last_unstale":"2024-03-16T11:05:03.111534+0000","last_undegraded":"2024-03-16T11:05:03.111534+0000","last_fullsized":"2024-03-16T11:05:03.111534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:07:51.828833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103117+0000","last_change":"2024-03-16T11:03:59.288822+0000","last_active":"2024-03-16T11:05:03.103117+0000","last_peered":"2024-03-16T11:05:03.103117+0000","last_clean":"2024-03-16T11:05:03.103117+0000","last_became_active":"2024-03-16T11:03:59.288516+0000","last_became_peered":"2024-03-16T11:03:59.288516+0000","last_unstale":"2024-03-16T11:05:03.103117+0000","last_undegraded":"2024-03-16T11:05:03.103117+0000","last_fullsized":"2024-03-16T11:05:03.103117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:32:38.862476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105721+0000","last_change":"2024-03-16T11:04:00.296117+0000","last_active":"2024-03-16T11:05:03.105721+0000","last_peered":"2024-03-16T11:05:03.105721+0000","last_clean":"2024-03-16T11:05:03.105721+0000","last_became_active":"2024-03-16T11:04:00.295664+0000","last_became_peered":"2024-03-16T11:04:00.295664+0000","last_unstale":"2024-03-16T11:05:03.105721+0000","last_undegraded":"2024-03-16T11:05:03.105721+0000","last_fullsized":"2024-03-16T11:05:03.105721+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.448616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105812+0000","last_change":"2024-03-16T11:04:00.295035+0000","last_active":"2024-03-16T11:05:03.105812+0000","last_peered":"2024-03-16T11:05:03.105812+0000","last_clean":"2024-03-16T11:05:03.105812+0000","last_became_active":"2024-03-16T11:04:00.294689+0000","last_became_peered":"2024-03-16T11:04:00.294689+0000","last_unstale":"2024-03-16T11:05:03.105812+0000","last_undegraded":"2024-03-16T11:05:03.105812+0000","last_fullsized":"2024-03-16T11:05:03.105812+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:55.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548695+0000","last_change":"2024-03-16T11:04:00.292831+0000","last_active":"2024-03-16T11:05:03.548695+0000","last_peered":"2024-03-16T11:05:03.548695+0000","last_clean":"2024-03-16T11:05:03.548695+0000","last_became_active":"2024-03-16T11:04:00.292387+0000","last_became_peered":"2024-03-16T11:04:00.292387+0000","last_unstale":"2024-03-16T11:05:03.548695+0000","last_undegraded":"2024-03-16T11:05:03.548695+0000","last_fullsized":"2024-03-16T11:05:03.548695+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:14:53.273535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548822+0000","last_change":"2024-03-16T11:04:00.291267+0000","last_active":"2024-03-16T11:05:03.548822+0000","last_peered":"2024-03-16T11:05:03.548822+0000","last_clean":"2024-03-16T11:05:03.548822+0000","last_became_active":"2024-03-16T11:04:00.290911+0000","last_became_peered":"2024-03-16T11:04:00.290911+0000","last_unstale":"2024-03-16T11:05:03.548822+0000","last_undegraded":"2024-03-16T11:05:03.548822+0000","last_fullsized":"2024-03-16T11:05:03.548822+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:35.285167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105324+0000","last_change":"2024-03-16T11:04:00.300470+0000","last_active":"2024-03-16T11:05:03.105324+0000","last_peered":"2024-03-16T11:05:03.105324+0000","last_clean":"2024-03-16T11:05:03.105324+0000","last_became_active":"2024-03-16T11:04:00.300107+0000","last_became_peered":"2024-03-16T11:04:00.300107+0000","last_unstale":"2024-03-16T11:05:03.105324+0000","last_undegraded":"2024-03-16T11:05:03.105324+0000","last_fullsized":"2024-03-16T11:05:03.105324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:18.718749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103549+0000","last_change":"2024-03-16T11:04:00.300857+0000","last_active":"2024-03-16T11:05:03.103549+0000","last_peered":"2024-03-16T11:05:03.103549+0000","last_clean":"2024-03-16T11:05:03.103549+0000","last_became_active":"2024-03-16T11:04:00.300551+0000","last_became_peered":"2024-03-16T11:04:00.300551+0000","last_unstale":"2024-03-16T11:05:03.103549+0000","last_undegraded":"2024-03-16T11:05:03.103549+0000","last_fullsized":"2024-03-16T11:05:03.103549+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:24.362561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.110781+0000","last_change":"2024-03-16T11:05:03.110781+0000","last_active":"2024-03-16T11:05:03.110781+0000","last_peered":"2024-03-16T11:05:03.110781+0000","last_clean":"2024-03-16T11:05:03.110781+0000","last_became_active":"2024-03-16T11:05:03.110375+0000","last_became_peered":"2024-03-16T11:05:03.110375+0000","last_unstale":"2024-03-16T11:05:03.110781+0000","last_undegraded":"2024-03-16T11:05:03.110781+0000","last_fullsized":"2024-03-16T11:05:03.110781+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:28.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.549286+0000","last_change":"2024-03-16T11:04:00.294943+0000","last_active":"2024-03-16T11:05:03.549286+0000","last_peered":"2024-03-16T11:05:03.549286+0000","last_clean":"2024-03-16T11:05:03.549286+0000","last_became_active":"2024-03-16T11:04:00.294637+0000","last_became_peered":"2024-03-16T11:04:00.294637+0000","last_unstale":"2024-03-16T11:05:03.549286+0000","last_undegraded":"2024-03-16T11:05:03.549286+0000","last_fullsized":"2024-03-16T11:05:03.549286+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:03:09.716753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107606+0000","last_change":"2024-03-16T11:04:00.300737+0000","last_active":"2024-03-16T11:05:03.107606+0000","last_peered":"2024-03-16T11:05:03.107606+0000","last_clean":"2024-03-16T11:05:03.107606+0000","last_became_active":"2024-03-16T11:04:00.300410+0000","last_became_peered":"2024-03-16T11:04:00.300410+0000","last_unstale":"2024-03-16T11:05:03.107606+0000","last_undegraded":"2024-03-16T11:05:03.107606+0000","last_fullsized":"2024-03-16T11:05:03.107606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:18.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102837+0000","last_change":"2024-03-16T11:04:00.292958+0000","last_active":"2024-03-16T11:05:03.102837+0000","last_peered":"2024-03-16T11:05:03.102837+0000","last_clean":"2024-03-16T11:05:03.102837+0000","last_became_active":"2024-03-16T11:04:00.292599+0000","last_became_peered":"2024-03-16T11:04:00.292599+0000","last_unstale":"2024-03-16T11:05:03.102837+0000","last_undegraded":"2024-03-16T11:05:03.102837+0000","last_fullsized":"2024-03-16T11:05:03.102837+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:46.370597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106664+0000","last_change":"2024-03-16T11:04:00.291381+0000","last_active":"2024-03-16T11:05:03.106664+0000","last_peered":"2024-03-16T11:05:03.106664+0000","last_clean":"2024-03-16T11:05:03.106664+0000","last_became_active":"2024-03-16T11:04:00.291046+0000","last_became_peered":"2024-03-16T11:04:00.291046+0000","last_unstale":"2024-03-16T11:05:03.106664+0000","last_undegraded":"2024-03-16T11:05:03.106664+0000","last_fullsized":"2024-03-16T11:05:03.106664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:08.185733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.101021+0000","last_change":"2024-03-16T11:04:00.301381+0000","last_active":"2024-03-16T11:05:03.101021+0000","last_peered":"2024-03-16T11:05:03.101021+0000","last_clean":"2024-03-16T11:05:03.101021+0000","last_became_active":"2024-03-16T11:04:00.300990+0000","last_became_peered":"2024-03-16T11:04:00.300990+0000","last_unstale":"2024-03-16T11:05:03.101021+0000","last_undegraded":"2024-03-16T11:05:03.101021+0000","last_fullsized":"2024-03-16T11:05:03.101021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:07:43.775066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103609+0000","last_change":"2024-03-16T11:04:00.298220+0000","last_active":"2024-03-16T11:05:03.103609+0000","last_peered":"2024-03-16T11:05:03.103609+0000","last_clean":"2024-03-16T11:05:03.103609+0000","last_became_active":"2024-03-16T11:04:00.297776+0000","last_became_peered":"2024-03-16T11:04:00.297776+0000","last_unstale":"2024-03-16T11:05:03.103609+0000","last_undegraded":"2024-03-16T11:05:03.103609+0000","last_fullsized":"2024-03-16T11:05:03.103609+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:40.387309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106051+0000","last_change":"2024-03-16T11:04:00.292988+0000","last_active":"2024-03-16T11:05:03.106051+0000","last_peered":"2024-03-16T11:05:03.106051+0000","last_clean":"2024-03-16T11:05:03.106051+0000","last_became_active":"2024-03-16T11:04:00.292655+0000","last_became_peered":"2024-03-16T11:04:00.292655+0000","last_unstale":"2024-03-16T11:05:03.106051+0000","last_undegraded":"2024-03-16T11:05:03.106051+0000","last_fullsized":"2024-03-16T11:05:03.106051+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:33:43.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62913,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":455,"num_read_kb":293,"num_write":154,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":188739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285596,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":471093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547234,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32824,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712328,"statfs":{"total":95995035648,"available":95961423872,"internally_reserved":0,"allocated":1679360,"data_stored":1121855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743463,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":471307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005099,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":471205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266736,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1063157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430262,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":470813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":529401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1007616,"data_stored":471967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":530751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":471007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":471524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691900,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1601536,"data_stored":1062333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:05:17.924 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-16T11:05:17.924 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T11:05:17.924 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-16T11:05:17.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-16T11:05:18.166 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:19 smithi098 ceph-mon[35848]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:19 smithi098 ceph-mon[35848]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:05:19.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:19 smithi078 ceph-mon[32235]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:19.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:19 smithi078 ceph-mon[32235]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:05:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:19 smithi153 ceph-mon[35521]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:19 smithi153 ceph-mon[35521]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:05:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 -- 172.21.15.78:0/591123520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 msgr2=0x7fbc1c1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/591123520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c1027d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009990 tx=0x7fbc0c009ca0 comp rx=0 tx=0).stop 2024-03-16T11:05:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 -- 172.21.15.78:0/591123520 shutdown_connections 2024-03-16T11:05:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/591123520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc1c1083a0 0x7fbc1c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/591123520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 0x7fbc1c10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/591123520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c1027d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 -- 172.21.15.78:0/591123520 >> 172.21.15.78:0/591123520 conn(0x7fbc1c079390 msgr2=0x7fbc1c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 -- 172.21.15.78:0/591123520 shutdown_connections 2024-03-16T11:05:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.835+0000 7fbc21ef8700 1 -- 172.21.15.78:0/591123520 wait complete. 2024-03-16T11:05:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 Processor -- start 2024-03-16T11:05:19.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 -- start start 2024-03-16T11:05:19.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:19.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 0x7fbc1c19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc1c1083a0 0x7fbc1c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c112450 con 0x7fbc1c102350 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c1122d0 con 0x7fbc1c102d10 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc21ef8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c1125d0 con 0x7fbc1c1083a0 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc1b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 0x7fbc1c19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc1b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42678/0 (socket says 172.21.15.78:42678) 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/1116361227 learned_addr learned my addr 172.21.15.78:0/1116361227 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 0x7fbc1c19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:34296/0 (socket says 172.21.15.78:34296) 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.836+0000 7fbc1bfff700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc1c1083a0 0x7fbc1c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/1116361227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc1c1083a0 msgr2=0x7fbc1c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc1b7fe700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc1c1083a0 0x7fbc1c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/1116361227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 msgr2=0x7fbc1c19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc1b7fe700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 0x7fbc1c19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc1b7fe700 1 -- 172.21.15.78:0/1116361227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c198f70 con 0x7fbc1c102350 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc1affd700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 0x7fbc1c19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc1b7fe700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c19e610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000e9c0 tx=0x7fbc1000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc18ff9700 1 -- 172.21.15.78:0/1116361227 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000cbd0 con 0x7fbc1c102350 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc18ff9700 1 -- 172.21.15.78:0/1116361227 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc1000ee50 con 0x7fbc1c102350 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc18ff9700 1 -- 172.21.15.78:0/1116361227 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc10005550 con 0x7fbc1c102350 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c199260 con 0x7fbc1c102350 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.837+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c1a5470 con 0x7fbc1c102350 2024-03-16T11:05:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.838+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c103a10 con 0x7fbc1c102350 2024-03-16T11:05:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.839+0000 7fbc18ff9700 1 -- 172.21.15.78:0/1116361227 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbc10004750 con 0x7fbc1c102350 2024-03-16T11:05:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.840+0000 7fbc18ff9700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbc040a29a0 0x7fbc040a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.840+0000 7fbc1affd700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbc040a29a0 0x7fbc040a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.840+0000 7fbc18ff9700 1 -- 172.21.15.78:0/1116361227 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbc100ced70 con 0x7fbc1c102350 2024-03-16T11:05:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.840+0000 7fbc1affd700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbc040a29a0 0x7fbc040a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c079a70 tx=0x7fbc0c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:19.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:19.844+0000 7fbc18ff9700 1 -- 172.21.15.78:0/1116361227 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc1009bcb0 con 0x7fbc1c102350 2024-03-16T11:05:20.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.083+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbc1c02db70 con 0x7fbc1c102350 2024-03-16T11:05:20.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.083+0000 7fbc18ff9700 1 -- 172.21.15.78:0/1116361227 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fbc1009b5b0 con 0x7fbc1c102350 2024-03-16T11:05:20.084 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:20.084 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T11:05:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.085+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbc040a29a0 msgr2=0x7fbc040a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.085+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbc040a29a0 0x7fbc040a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c079a70 tx=0x7fbc0c01a040 comp rx=0 tx=0).stop 2024-03-16T11:05:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.085+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 msgr2=0x7fbc1c19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.085+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c19e610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000e9c0 tx=0x7fbc1000ecd0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 shutdown_connections 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fbc040a29a0 0x7fbc040a4e60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc1c1083a0 0x7fbc1c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc1c102350 0x7fbc1c19e610 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 --2- 172.21.15.78:0/1116361227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc1c102d10 0x7fbc1c19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 >> 172.21.15.78:0/1116361227 conn(0x7fbc1c079390 msgr2=0x7fbc1c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 shutdown_connections 2024-03-16T11:05:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.086+0000 7fbc21ef8700 1 -- 172.21.15.78:0/1116361227 wait complete. 2024-03-16T11:05:20.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:20 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1116361227' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T11:05:20.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:20 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1116361227' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T11:05:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:20 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1116361227' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T11:05:20.760 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-16T11:05:20.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 -- 172.21.15.78:0/3374314240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101d90 msgr2=0x7fd2381021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 --2- 172.21.15.78:0/3374314240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101d90 0x7fd2381021b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009900 tx=0x7fd22c02bc80 comp rx=0 tx=0).stop 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 -- 172.21.15.78:0/3374314240 shutdown_connections 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 --2- 172.21.15.78:0/3374314240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd238101310 0x7fd238108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 --2- 172.21.15.78:0/3374314240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238100950 0x7fd238100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 --2- 172.21.15.78:0/3374314240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101d90 0x7fd2381021b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 -- 172.21.15.78:0/3374314240 >> 172.21.15.78:0/3374314240 conn(0x7fd2380fb400 msgr2=0x7fd2380fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 -- 172.21.15.78:0/3374314240 shutdown_connections 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 -- 172.21.15.78:0/3374314240 wait complete. 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 Processor -- start 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 -- start start 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd238100950 0x7fd238199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101310 0x7fd23819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.864+0000 7fd23f227640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238101d90 0x7fd2381a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd23f227640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd23810d390 con 0x7fd238101310 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd23f227640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd23810d210 con 0x7fd238100950 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd23f227640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd23810d510 con 0x7fd238101d90 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101310 0x7fd23819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101310 0x7fd23819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42692/0 (socket says 172.21.15.78:42692) 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 -- 172.21.15.78:0/2128753372 learned_addr learned my addr 172.21.15.78:0/2128753372 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd23cfc3640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd238100950 0x7fd238199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd23d7c4640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238101d90 0x7fd2381a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 -- 172.21.15.78:0/2128753372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238101d90 msgr2=0x7fd2381a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238101d90 0x7fd2381a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 -- 172.21.15.78:0/2128753372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd238100950 msgr2=0x7fd238199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd238100950 0x7fd238199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 -- 172.21.15.78:0/2128753372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2381a18b0 con 0x7fd238101310 2024-03-16T11:05:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd237fff640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101310 0x7fd23819a1b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd22400e9c0 tx=0x7fd22400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd235ffb640 1 -- 172.21.15.78:0/2128753372 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22400cd80 con 0x7fd238101310 2024-03-16T11:05:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd235ffb640 1 -- 172.21.15.78:0/2128753372 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2240035f0 con 0x7fd238101310 2024-03-16T11:05:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2381a1ba0 con 0x7fd238101310 2024-03-16T11:05:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd235ffb640 1 -- 172.21.15.78:0/2128753372 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd224010690 con 0x7fd238101310 2024-03-16T11:05:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.865+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2381a20f0 con 0x7fd238101310 2024-03-16T11:05:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.866+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2381a2320 con 0x7fd238101310 2024-03-16T11:05:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.867+0000 7fd235ffb640 1 -- 172.21.15.78:0/2128753372 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd2240037a0 con 0x7fd238101310 2024-03-16T11:05:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.868+0000 7fd235ffb640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd2280a2910 0x7fd2280a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.868+0000 7fd23cfc3640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd2280a2910 0x7fd2280a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.868+0000 7fd235ffb640 1 -- 172.21.15.78:0/2128753372 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd2240cefd0 con 0x7fd238101310 2024-03-16T11:05:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.868+0000 7fd23cfc3640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd2280a2910 0x7fd2280a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd23819ab40 tx=0x7fd22c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:20.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:20.872+0000 7fd235ffb640 1 -- 172.21.15.78:0/2128753372 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd22409bfd0 con 0x7fd238101310 2024-03-16T11:05:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.046+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2381a2590 con 0x7fd238101310 2024-03-16T11:05:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.048+0000 7fd235ffb640 1 -- 172.21.15.78:0/2128753372 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7fd22409b8d0 con 0x7fd238101310 2024-03-16T11:05:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:21.049 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd2280a2910 msgr2=0x7fd2280a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd2280a2910 0x7fd2280a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd23819ab40 tx=0x7fd22c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101310 msgr2=0x7fd23819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101310 0x7fd23819a1b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd22400e9c0 tx=0x7fd22400ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 shutdown_connections 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd2280a2910 0x7fd2280a4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238101d90 0x7fd2381a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd238101310 0x7fd23819a1b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 --2- 172.21.15.78:0/2128753372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd238100950 0x7fd238199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 >> 172.21.15.78:0/2128753372 conn(0x7fd2380fb400 msgr2=0x7fd238107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 shutdown_connections 2024-03-16T11:05:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.050+0000 7fd23f227640 1 -- 172.21.15.78:0/2128753372 wait complete. 2024-03-16T11:05:21.062 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:05:21.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:05:21.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.168+0000 7f11cf8f1640 1 -- 172.21.15.78:0/258485497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c8107540 msgr2=0x7f11c8109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.168+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/258485497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c8107540 0x7f11c8109930 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009ca0 tx=0x7f11c402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- 172.21.15.78:0/258485497 shutdown_connections 2024-03-16T11:05:21.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/258485497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c8107540 0x7f11c8109930 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/258485497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8104c10 0x7f11c8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/258485497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c81022e0 0x7f11c81046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- 172.21.15.78:0/258485497 >> 172.21.15.78:0/258485497 conn(0x7f11c80fbc70 msgr2=0x7f11c80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- 172.21.15.78:0/258485497 shutdown_connections 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- 172.21.15.78:0/258485497 wait complete. 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 Processor -- start 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- start start 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c81022e0 0x7f11c8101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8104c10 0x7f11c8100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c8107540 0x7f11c81007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f11c810d330 con 0x7f11c81022e0 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11c810d1b0 con 0x7f11c8107540 2024-03-16T11:05:21.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cf8f1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11c810d4b0 con 0x7f11c8104c10 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cd68d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c81022e0 0x7f11c8101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cd68d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c81022e0 0x7f11c8101c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42702/0 (socket says 172.21.15.78:42702) 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.169+0000 7f11cd68d640 1 -- 172.21.15.78:0/3696051312 learned_addr learned my addr 172.21.15.78:0/3696051312 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cde8e640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c8107540 0x7f11c81007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cce8c640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8104c10 0x7f11c8100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cd68d640 1 -- 172.21.15.78:0/3696051312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8104c10 msgr2=0x7f11c8100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cd68d640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8104c10 0x7f11c8100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cd68d640 1 -- 172.21.15.78:0/3696051312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c8107540 msgr2=0x7f11c81007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cd68d640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c8107540 0x7f11c81007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cd68d640 1 -- 172.21.15.78:0/3696051312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c8101060 con 0x7f11c81022e0 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cce8c640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8104c10 0x7f11c8100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cd68d640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c81022e0 0x7f11c8101c50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f11c000ea60 tx=0x7f11c000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11be7fc640 1 -- 172.21.15.78:0/3696051312 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c000cde0 con 0x7f11c81022e0 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11be7fc640 1 -- 172.21.15.78:0/3696051312 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11c00035f0 con 0x7f11c81022e0 2024-03-16T11:05:21.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c81a1b60 con 0x7f11c81022e0 2024-03-16T11:05:21.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11be7fc640 1 -- 172.21.15.78:0/3696051312 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0010690 con 0x7f11c81022e0 2024-03-16T11:05:21.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.170+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c81a20b0 con 0x7f11c81022e0 2024-03-16T11:05:21.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.171+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c80fd760 con 0x7f11c81022e0 2024-03-16T11:05:21.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.172+0000 7f11be7fc640 1 -- 172.21.15.78:0/3696051312 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f11c00037a0 con 0x7f11c81022e0 2024-03-16T11:05:21.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.172+0000 7f11be7fc640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f11b40a2890 0x7f11b40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.173+0000 7f11cce8c640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f11b40a2890 0x7f11b40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.173+0000 7f11be7fc640 1 -- 172.21.15.78:0/3696051312 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f11c00cef80 con 0x7f11c81022e0 2024-03-16T11:05:21.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.173+0000 7f11cce8c640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f11b40a2890 0x7f11b40a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f11c81019c0 tx=0x7f11b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:21.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.177+0000 7f11be7fc640 1 -- 172.21.15.78:0/3696051312 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11c009bec0 con 0x7f11c81022e0 2024-03-16T11:05:21.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.349+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f11c802d380 con 0x7f11c81022e0 2024-03-16T11:05:21.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.350+0000 7f11be7fc640 1 -- 172.21.15.78:0/3696051312 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f11c009b7c0 con 0x7f11c81022e0 2024-03-16T11:05:21.351 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:05:21.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f11b40a2890 msgr2=0x7f11b40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f11b40a2890 0x7f11b40a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f11c81019c0 tx=0x7f11b8005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c81022e0 msgr2=0x7f11c8101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c81022e0 0x7f11c8101c50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f11c000ea60 tx=0x7f11c000ef20 comp rx=0 tx=0).stop 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 shutdown_connections 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c8107540 0x7f11c81007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f11b40a2890 0x7f11b40a4d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8104c10 0x7f11c8100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 --2- 172.21.15.78:0/3696051312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11c81022e0 0x7f11c8101c50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 >> 172.21.15.78:0/3696051312 conn(0x7f11c80fbc70 msgr2=0x7f11c8102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 shutdown_connections 2024-03-16T11:05:21.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.352+0000 7f11cf8f1640 1 -- 172.21.15.78:0/3696051312 wait complete. 2024-03-16T11:05:21.362 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:21 smithi078 ceph-mon[32235]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:21.362 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:21 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2128753372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:21.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:05:21.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:21 smithi098 ceph-mon[35848]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:21.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:21 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2128753372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 -- 172.21.15.78:0/884015339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01018e0 msgr2=0x7f74a0101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 --2- 172.21.15.78:0/884015339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01018e0 0x7f74a0101d60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7490009900 tx=0x7f749002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 -- 172.21.15.78:0/884015339 shutdown_connections 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 --2- 172.21.15.78:0/884015339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a01022a0 0x7f74a0108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 --2- 172.21.15.78:0/884015339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01018e0 0x7f74a0101d60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 --2- 172.21.15.78:0/884015339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a01006f0 0x7f74a0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 -- 172.21.15.78:0/884015339 >> 172.21.15.78:0/884015339 conn(0x7f74a00fbc80 msgr2=0x7f74a00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 -- 172.21.15.78:0/884015339 shutdown_connections 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.468+0000 7f74a7f76640 1 -- 172.21.15.78:0/884015339 wait complete. 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 Processor -- start 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 -- start start 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01006f0 0x7f74a006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a01018e0 0x7f74a019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a01022a0 0x7f74a019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f74a010c220 con 0x7f74a01006f0 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74a010c0a0 con 0x7f74a01018e0 2024-03-16T11:05:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a7f76640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74a010c3a0 con 0x7f74a01022a0 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01006f0 0x7f74a006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01006f0 0x7f74a006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42720/0 (socket says 172.21.15.78:42720) 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 -- 172.21.15.78:0/3830317453 learned_addr learned my addr 172.21.15.78:0/3830317453 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a6513640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a01022a0 0x7f74a019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5511640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a01018e0 0x7f74a019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 -- 172.21.15.78:0/3830317453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a01022a0 msgr2=0x7f74a019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a01022a0 0x7f74a019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 -- 172.21.15.78:0/3830317453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a01018e0 msgr2=0x7f74a019c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a01018e0 0x7f74a019c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 -- 172.21.15.78:0/3830317453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a019f6a0 con 0x7f74a01006f0 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5511640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a01018e0 0x7f74a019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f74a5d12640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01006f0 0x7f74a006a960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f749c00e9c0 tx=0x7f749c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.469+0000 7f7496ffd640 1 -- 172.21.15.78:0/3830317453 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c00ccc0 con 0x7f74a01006f0 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.470+0000 7f7496ffd640 1 -- 172.21.15.78:0/3830317453 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f749c0035f0 con 0x7f74a01006f0 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.470+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a019f990 con 0x7f74a01006f0 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.470+0000 7f7496ffd640 1 -- 172.21.15.78:0/3830317453 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c00ce60 con 0x7f74a01006f0 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.470+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a019fe90 con 0x7f74a01006f0 2024-03-16T11:05:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.471+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a0107ad0 con 0x7f74a01006f0 2024-03-16T11:05:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.472+0000 7f7496ffd640 1 -- 172.21.15.78:0/3830317453 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f749c0037a0 con 0x7f74a01006f0 2024-03-16T11:05:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.472+0000 7f7496ffd640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f748c0a2890 0x7f748c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.472+0000 7f74a5511640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f748c0a2890 0x7f748c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.472+0000 7f7496ffd640 1 -- 172.21.15.78:0/3830317453 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f749c0cf190 con 0x7f74a01006f0 2024-03-16T11:05:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.473+0000 7f74a5511640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f748c0a2890 0x7f748c0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f74a006a6d0 tx=0x7f749002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.477+0000 7f7496ffd640 1 -- 172.21.15.78:0/3830317453 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f749c09c0d0 con 0x7f74a01006f0 2024-03-16T11:05:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:21 smithi153 ceph-mon[35521]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:21.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:21 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2128753372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:21.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.650+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74a002d3b0 con 0x7f74a01006f0 2024-03-16T11:05:21.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.651+0000 7f7496ffd640 1 -- 172.21.15.78:0/3830317453 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f749c09b9d0 con 0x7f74a01006f0 2024-03-16T11:05:21.651 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.652+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f748c0a2890 msgr2=0x7f748c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.652+0000 7f74a7f76640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f748c0a2890 0x7f748c0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f74a006a6d0 tx=0x7f749002c7e0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.652+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01006f0 msgr2=0x7f74a006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.652+0000 7f74a7f76640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01006f0 0x7f74a006a960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f749c00e9c0 tx=0x7f749c00edc0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 shutdown_connections 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f748c0a2890 0x7f748c0a4d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a01022a0 0x7f74a019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74a01018e0 0x7f74a019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 --2- 172.21.15.78:0/3830317453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74a01006f0 0x7f74a006a960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 >> 172.21.15.78:0/3830317453 conn(0x7f74a00fbc80 msgr2=0x7f74a0104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 shutdown_connections 2024-03-16T11:05:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.653+0000 7f74a7f76640 1 -- 172.21.15.78:0/3830317453 wait complete. 2024-03-16T11:05:21.665 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:05:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.772+0000 7efe353b9640 1 -- 172.21.15.78:0/1188971198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 msgr2=0x7efe30100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.772+0000 7efe353b9640 1 --2- 172.21.15.78:0/1188971198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30100b10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efe20009900 tx=0x7efe2002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- 172.21.15.78:0/1188971198 shutdown_connections 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 --2- 172.21.15.78:0/1188971198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe301022a0 0x7efe30108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 --2- 172.21.15.78:0/1188971198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe301018e0 0x7efe30101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 --2- 172.21.15.78:0/1188971198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30100b10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- 172.21.15.78:0/1188971198 >> 172.21.15.78:0/1188971198 conn(0x7efe300fbc80 msgr2=0x7efe300fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- 172.21.15.78:0/1188971198 shutdown_connections 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- 172.21.15.78:0/1188971198 wait complete. 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 Processor -- start 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- start start 2024-03-16T11:05:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe301018e0 0x7efe30197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe301022a0 0x7efe3019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efe3010d300 con 0x7efe301006f0 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe3010d180 con 0x7efe301022a0 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.773+0000 7efe353b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe3010d480 con 0x7efe301018e0 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42736/0 (socket says 172.21.15.78:42736) 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 -- 172.21.15.78:0/3509404910 learned_addr learned my addr 172.21.15.78:0/3509404910 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2f7fe640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe301022a0 0x7efe3019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2e7fc640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe301018e0 0x7efe30197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 -- 172.21.15.78:0/3509404910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe301018e0 msgr2=0x7efe30197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe301018e0 0x7efe30197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 -- 172.21.15.78:0/3509404910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe301022a0 msgr2=0x7efe3019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe301022a0 0x7efe3019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 -- 172.21.15.78:0/3509404910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3019f6c0 con 0x7efe301006f0 2024-03-16T11:05:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe2effd640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30197aa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efe20000c00 tx=0x7efe20004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe17fff640 1 -- 172.21.15.78:0/3509404910 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20043020 con 0x7efe301006f0 2024-03-16T11:05:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3019f950 con 0x7efe301006f0 2024-03-16T11:05:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe17fff640 1 -- 172.21.15.78:0/3509404910 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe20004440 con 0x7efe301006f0 2024-03-16T11:05:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe17fff640 1 -- 172.21.15.78:0/3509404910 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20003c90 con 0x7efe301006f0 2024-03-16T11:05:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.774+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe3019fea0 con 0x7efe301006f0 2024-03-16T11:05:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.775+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe30107b80 con 0x7efe301006f0 2024-03-16T11:05:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.776+0000 7efe17fff640 1 -- 172.21.15.78:0/3509404910 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efe20046020 con 0x7efe301006f0 2024-03-16T11:05:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.777+0000 7efe17fff640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe180a2890 0x7efe180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.777+0000 7efe2e7fc640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe180a2890 0x7efe180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.777+0000 7efe17fff640 1 -- 172.21.15.78:0/3509404910 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7efe200f0c80 con 0x7efe301006f0 2024-03-16T11:05:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.777+0000 7efe2e7fc640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe180a2890 0x7efe180a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efe30198fb0 tx=0x7efe24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.781+0000 7efe17fff640 1 -- 172.21.15.78:0/3509404910 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe2004d020 con 0x7efe301006f0 2024-03-16T11:05:21.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.955+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efe3002d3d0 con 0x7efe301006f0 2024-03-16T11:05:21.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.955+0000 7efe17fff640 1 -- 172.21.15.78:0/3509404910 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7efe200bdcd0 con 0x7efe301006f0 2024-03-16T11:05:21.956 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:05:21.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe180a2890 msgr2=0x7efe180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe180a2890 0x7efe180a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efe30198fb0 tx=0x7efe24005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:21.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 msgr2=0x7efe30197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30197aa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efe20000c00 tx=0x7efe20004040 comp rx=0 tx=0).stop 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 shutdown_connections 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe301022a0 0x7efe3019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe180a2890 0x7efe180a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe301018e0 0x7efe30197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 --2- 172.21.15.78:0/3509404910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301006f0 0x7efe30197aa0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 >> 172.21.15.78:0/3509404910 conn(0x7efe300fbc80 msgr2=0x7efe30104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 shutdown_connections 2024-03-16T11:05:21.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:21.957+0000 7efe353b9640 1 -- 172.21.15.78:0/3509404910 wait complete. 2024-03-16T11:05:21.970 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:05:22.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 -- 172.21.15.78:0/952556730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901022b0 msgr2=0x7f1090108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 --2- 172.21.15.78:0/952556730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901022b0 0x7f1090108b40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1088009ca0 tx=0x7f108802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 -- 172.21.15.78:0/952556730 shutdown_connections 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 --2- 172.21.15.78:0/952556730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901022b0 0x7f1090108b40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 --2- 172.21.15.78:0/952556730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10901018f0 0x7f1090101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 --2- 172.21.15.78:0/952556730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1090100700 0x7f1090100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 -- 172.21.15.78:0/952556730 >> 172.21.15.78:0/952556730 conn(0x7f10900fbc70 msgr2=0x7f10900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 -- 172.21.15.78:0/952556730 shutdown_connections 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 -- 172.21.15.78:0/952556730 wait complete. 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.072+0000 7f109831b640 1 Processor -- start 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 -- start start 2024-03-16T11:05:22.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1090100700 0x7f1090104760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901018f0 0x7f1090102df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10901022b0 0x7f1090103330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f109010c230 con 0x7f10901018f0 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f109010c0b0 con 0x7f10901022b0 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f109010c3b0 con 0x7f1090100700 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901018f0 0x7f1090102df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901018f0 0x7f1090102df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42752/0 (socket says 172.21.15.78:42752) 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 -- 172.21.15.78:0/124184867 learned_addr learned my addr 172.21.15.78:0/124184867 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10968b8640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10901022b0 0x7f1090103330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10960b7640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1090100700 0x7f1090104760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 -- 172.21.15.78:0/124184867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1090100700 msgr2=0x7f1090104760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1090100700 0x7f1090104760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 -- 172.21.15.78:0/124184867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10901022b0 msgr2=0x7f1090103330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10901022b0 0x7f1090103330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 -- 172.21.15.78:0/124184867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1090103bf0 con 0x7f10901018f0 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10958b6640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901018f0 0x7f1090102df0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f108400ea60 tx=0x7f108400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10837fe640 1 -- 172.21.15.78:0/124184867 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108400cde0 con 0x7f10901018f0 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10837fe640 1 -- 172.21.15.78:0/124184867 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10840035f0 con 0x7f10901018f0 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10901a1b30 con 0x7f10901018f0 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f10837fe640 1 -- 172.21.15.78:0/124184867 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084010690 con 0x7f10901018f0 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.073+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10901a2050 con 0x7f10901018f0 2024-03-16T11:05:22.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.074+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10901a2280 con 0x7f10901018f0 2024-03-16T11:05:22.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.075+0000 7f10837fe640 1 -- 172.21.15.78:0/124184867 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f10840037a0 con 0x7f10901018f0 2024-03-16T11:05:22.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.076+0000 7f10837fe640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f107c0a2890 0x7f107c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.076+0000 7f10960b7640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f107c0a2890 0x7f107c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.076+0000 7f10837fe640 1 -- 172.21.15.78:0/124184867 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f10840cefb0 con 0x7f10901018f0 2024-03-16T11:05:22.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.076+0000 7f10960b7640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f107c0a2890 0x7f107c0a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1090101750 tx=0x7f108c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:22.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.080+0000 7f10837fe640 1 -- 172.21.15.78:0/124184867 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1084053740 con 0x7f10901018f0 2024-03-16T11:05:22.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.292+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10901a2570 con 0x7f10901018f0 2024-03-16T11:05:22.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.293+0000 7f10837fe640 1 -- 172.21.15.78:0/124184867 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11869 (secure 0 0 0) 0x7f108409bfb0 con 0x7f10901018f0 2024-03-16T11:05:22.293 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:22.293 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:04:03.658334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24488,24505,24521]},"id":1}]} 2024-03-16T11:05:22.294 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 21 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f107c0a2890 msgr2=0x7f107c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f107c0a2890 0x7f107c0a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1090101750 tx=0x7f108c005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901018f0 msgr2=0x7f1090102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901018f0 0x7f1090102df0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f108400ea60 tx=0x7f108400ef20 comp rx=0 tx=0).stop 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 shutdown_connections 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10901022b0 0x7f1090103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10901018f0 0x7f1090102df0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f107c0a2890 0x7f107c0a4d50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 --2- 172.21.15.78:0/124184867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1090100700 0x7f1090104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 >> 172.21.15.78:0/124184867 conn(0x7f10900fbc70 msgr2=0x7f1090069e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 shutdown_connections 2024-03-16T11:05:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.295+0000 7f109831b640 1 -- 172.21.15.78:0/124184867 wait complete. 2024-03-16T11:05:22.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:22 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3696051312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:22 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3830317453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:22 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3509404910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:22.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 -- 172.21.15.78:0/1798861927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef01022b0 msgr2=0x7fcef0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 --2- 172.21.15.78:0/1798861927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef01022b0 0x7fcef0108b40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009ca0 tx=0x7fcee802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 -- 172.21.15.78:0/1798861927 shutdown_connections 2024-03-16T11:05:22.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 --2- 172.21.15.78:0/1798861927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef01022b0 0x7fcef0108b40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 --2- 172.21.15.78:0/1798861927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcef01018f0 0x7fcef0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 --2- 172.21.15.78:0/1798861927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0100700 0x7fcef0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 -- 172.21.15.78:0/1798861927 >> 172.21.15.78:0/1798861927 conn(0x7fcef00fbc70 msgr2=0x7fcef00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.408+0000 7fcef6497640 1 -- 172.21.15.78:0/1798861927 shutdown_connections 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 -- 172.21.15.78:0/1798861927 wait complete. 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 Processor -- start 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 -- start start 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef0100700 0x7fcef01979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01018f0 0x7fcef0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcef01022b0 0x7fcef019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcef010d350 con 0x7fcef0100700 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcef010d1d0 con 0x7fcef01018f0 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fcef6497640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcef010d4d0 con 0x7fcef01022b0 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fceeffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef0100700 0x7fcef01979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fceeffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef0100700 0x7fcef01979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42764/0 (socket says 172.21.15.78:42764) 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fceeffff640 1 -- 172.21.15.78:0/862389513 learned_addr learned my addr 172.21.15.78:0/862389513 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:22.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.409+0000 7fceef7fe640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01018f0 0x7fcef0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fcef4a34640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcef01022b0 0x7fcef019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceeffff640 1 -- 172.21.15.78:0/862389513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcef01022b0 msgr2=0x7fcef019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceeffff640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcef01022b0 0x7fcef019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceeffff640 1 -- 172.21.15.78:0/862389513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01018f0 msgr2=0x7fcef0197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceeffff640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01018f0 0x7fcef0197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceeffff640 1 -- 172.21.15.78:0/862389513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef019f610 con 0x7fcef0100700 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fcef4a34640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcef01022b0 0x7fcef019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceeffff640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef0100700 0x7fcef01979f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcee000e9c0 tx=0x7fcee000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceed7fa640 1 -- 172.21.15.78:0/862389513 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee000cd80 con 0x7fcef0100700 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceed7fa640 1 -- 172.21.15.78:0/862389513 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcee00035f0 con 0x7fcef0100700 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef019f900 con 0x7fcef0100700 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fceed7fa640 1 -- 172.21.15.78:0/862389513 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0010690 con 0x7fcef0100700 2024-03-16T11:05:22.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.410+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef019fea0 con 0x7fcef0100700 2024-03-16T11:05:22.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.411+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef0107a20 con 0x7fcef0100700 2024-03-16T11:05:22.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.412+0000 7fceed7fa640 1 -- 172.21.15.78:0/862389513 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fcee00037a0 con 0x7fcef0100700 2024-03-16T11:05:22.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.413+0000 7fceed7fa640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fced80a2910 0x7fced80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.413+0000 7fceef7fe640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fced80a2910 0x7fced80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.413+0000 7fceed7fa640 1 -- 172.21.15.78:0/862389513 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcee00cefb0 con 0x7fcef0100700 2024-03-16T11:05:22.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.413+0000 7fceef7fe640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fced80a2910 0x7fced80a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcef0198f00 tx=0x7fcee4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:22.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.417+0000 7fceed7fa640 1 -- 172.21.15.78:0/862389513 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcee009bef0 con 0x7fcef0100700 2024-03-16T11:05:22.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:22 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3696051312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:22 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3830317453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:22 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3509404910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:22 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3696051312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:22 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3830317453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:22 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3509404910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.595+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcef00623e0 con 0x7fcef0100700 2024-03-16T11:05:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.597+0000 7fceed7fa640 1 -- 172.21.15.78:0/862389513 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7fcee009b7f0 con 0x7fcef0100700 2024-03-16T11:05:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:22.597 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.598+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fced80a2910 msgr2=0x7fced80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.598+0000 7fcef6497640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fced80a2910 0x7fced80a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcef0198f00 tx=0x7fcee4005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.598+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef0100700 msgr2=0x7fcef01979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.598+0000 7fcef6497640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef0100700 0x7fcef01979f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcee000e9c0 tx=0x7fcee000ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 shutdown_connections 2024-03-16T11:05:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fced80a2910 0x7fced80a4dd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcef01022b0 0x7fcef019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef01018f0 0x7fcef0197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 --2- 172.21.15.78:0/862389513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcef0100700 0x7fcef01979f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 >> 172.21.15.78:0/862389513 conn(0x7fcef00fbc70 msgr2=0x7fcef0104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:22.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 shutdown_connections 2024-03-16T11:05:22.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.599+0000 7fcef6497640 1 -- 172.21.15.78:0/862389513 wait complete. 2024-03-16T11:05:22.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:05:22.611 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:05:22.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 -- 172.21.15.78:0/3259092646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 msgr2=0x7f15701082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 --2- 172.21.15.78:0/3259092646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f15701082b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1568009ca0 tx=0x7f156802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 -- 172.21.15.78:0/3259092646 shutdown_connections 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 --2- 172.21.15.78:0/3259092646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f15701082b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 --2- 172.21.15.78:0/3259092646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1570100fd0 0x7f1570101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 --2- 172.21.15.78:0/3259092646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15700ffe80 0x7f15701002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 -- 172.21.15.78:0/3259092646 >> 172.21.15.78:0/3259092646 conn(0x7f15700fb410 msgr2=0x7f15700fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 -- 172.21.15.78:0/3259092646 shutdown_connections 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 -- 172.21.15.78:0/3259092646 wait complete. 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 Processor -- start 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.716+0000 7f1577af1640 1 -- start start 2024-03-16T11:05:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15700ffe80 0x7f15700fd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1570100fd0 0x7f157019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f157019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f157010b9c0 con 0x7f1570101990 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f157010b840 con 0x7f1570100fd0 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f157010bb40 con 0x7f15700ffe80 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f157019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f157019ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42776/0 (socket says 172.21.15.78:42776) 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 -- 172.21.15.78:0/3912440514 learned_addr learned my addr 172.21.15.78:0/3912440514 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157588d640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15700ffe80 0x7f15700fd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157508c640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1570100fd0 0x7f157019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 -- 172.21.15.78:0/3912440514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15700ffe80 msgr2=0x7f15700fd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15700ffe80 0x7f15700fd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 -- 172.21.15.78:0/3912440514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1570100fd0 msgr2=0x7f157019c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1570100fd0 0x7f157019c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 -- 172.21.15.78:0/3912440514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157019f5e0 con 0x7f1570101990 2024-03-16T11:05:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f157608e640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f157019ed20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f156802c700 tx=0x7f15680328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1562ffd640 1 -- 172.21.15.78:0/3912440514 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1568041020 con 0x7f1570101990 2024-03-16T11:05:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157019f870 con 0x7f1570101990 2024-03-16T11:05:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1562ffd640 1 -- 172.21.15.78:0/3912440514 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1568003e20 con 0x7f1570101990 2024-03-16T11:05:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1562ffd640 1 -- 172.21.15.78:0/3912440514 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1568039040 con 0x7f1570101990 2024-03-16T11:05:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.717+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f157019fdb0 con 0x7f1570101990 2024-03-16T11:05:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.718+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15701075c0 con 0x7f1570101990 2024-03-16T11:05:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.719+0000 7f1562ffd640 1 -- 172.21.15.78:0/3912440514 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1568044020 con 0x7f1570101990 2024-03-16T11:05:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.720+0000 7f1562ffd640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f155c0a2910 0x7f155c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.720+0000 7f157588d640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f155c0a2910 0x7f155c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.720+0000 7f1562ffd640 1 -- 172.21.15.78:0/3912440514 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f15680f0cb0 con 0x7f1570101990 2024-03-16T11:05:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.721+0000 7f157588d640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f155c0a2910 0x7f155c0a4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f15640095f0 tx=0x7f1564005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:22.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.725+0000 7f1562ffd640 1 -- 172.21.15.78:0/3912440514 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f156804b020 con 0x7f1570101990 2024-03-16T11:05:22.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.904+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15700623e0 con 0x7f1570101990 2024-03-16T11:05:22.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.904+0000 7f1562ffd640 1 -- 172.21.15.78:0/3912440514 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f15680bdcc0 con 0x7f1570101990 2024-03-16T11:05:22.905 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:05:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f155c0a2910 msgr2=0x7f155c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f155c0a2910 0x7f155c0a4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f15640095f0 tx=0x7f1564005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 msgr2=0x7f157019ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f157019ed20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f156802c700 tx=0x7f15680328a0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 shutdown_connections 2024-03-16T11:05:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570101990 0x7f157019ed20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1570100fd0 0x7f157019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f155c0a2910 0x7f155c0a4dd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 --2- 172.21.15.78:0/3912440514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15700ffe80 0x7f15700fd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:22.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 >> 172.21.15.78:0/3912440514 conn(0x7f15700fb410 msgr2=0x7f1570104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:22.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 shutdown_connections 2024-03-16T11:05:22.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:22.906+0000 7f1577af1640 1 -- 172.21.15.78:0/3912440514 wait complete. 2024-03-16T11:05:22.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:05:23.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.021+0000 7f82dbe75640 1 -- 172.21.15.78:0/862025504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41006e0 msgr2=0x7f82d4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.021+0000 7f82dbe75640 1 --2- 172.21.15.78:0/862025504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41006e0 0x7f82d4100b00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009900 tx=0x7f82d002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:23.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.021+0000 7f82dbe75640 1 -- 172.21.15.78:0/862025504 shutdown_connections 2024-03-16T11:05:23.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.021+0000 7f82dbe75640 1 --2- 172.21.15.78:0/862025504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82d4102290 0x7f82d4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 --2- 172.21.15.78:0/862025504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d41018d0 0x7f82d4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 --2- 172.21.15.78:0/862025504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41006e0 0x7f82d4100b00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 -- 172.21.15.78:0/862025504 >> 172.21.15.78:0/862025504 conn(0x7f82d40fbc70 msgr2=0x7f82d40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 -- 172.21.15.78:0/862025504 shutdown_connections 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 -- 172.21.15.78:0/862025504 wait complete. 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 Processor -- start 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 -- start start 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d41006e0 0x7f82d4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41018d0 0x7f82d4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82d4102290 0x7f82d419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f82d410c1e0 con 0x7f82d41018d0 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82d410c060 con 0x7f82d4102290 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82dbe75640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82d410c360 con 0x7f82d41006e0 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82d9410640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41018d0 0x7f82d4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82d9410640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41018d0 0x7f82d4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42808/0 (socket says 172.21.15.78:42808) 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82d9410640 1 -- 172.21.15.78:0/3469350385 learned_addr learned my addr 172.21.15.78:0/3469350385 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82d9c11640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d41006e0 0x7f82d4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.022+0000 7f82da412640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82d4102290 0x7f82d419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82d9410640 1 -- 172.21.15.78:0/3469350385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d41006e0 msgr2=0x7f82d4197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82d9410640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d41006e0 0x7f82d4197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82d9410640 1 -- 172.21.15.78:0/3469350385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82d4102290 msgr2=0x7f82d419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82d9410640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82d4102290 0x7f82d419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82d9410640 1 -- 172.21.15.78:0/3469350385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d419f6c0 con 0x7f82d41018d0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82d9410640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41018d0 0x7f82d4197fe0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f82c400e9f0 tx=0x7f82c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82caffd640 1 -- 172.21.15.78:0/3469350385 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c400ccc0 con 0x7f82d41018d0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82caffd640 1 -- 172.21.15.78:0/3469350385 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82c40108e0 con 0x7f82d41018d0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82caffd640 1 -- 172.21.15.78:0/3469350385 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c40187c0 con 0x7f82d41018d0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d419f9b0 con 0x7f82d41018d0 2024-03-16T11:05:23.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.023+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d419ff00 con 0x7f82d41018d0 2024-03-16T11:05:23.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.024+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d41a0130 con 0x7f82d41018d0 2024-03-16T11:05:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.025+0000 7f82caffd640 1 -- 172.21.15.78:0/3469350385 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f82c4010a90 con 0x7f82d41018d0 2024-03-16T11:05:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.025+0000 7f82caffd640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82c00a2890 0x7f82c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.026+0000 7f82d9c11640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82c00a2890 0x7f82c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.026+0000 7f82caffd640 1 -- 172.21.15.78:0/3469350385 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f82c40cefe0 con 0x7f82d41018d0 2024-03-16T11:05:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.026+0000 7f82d9c11640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82c00a2890 0x7f82c00a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f82d0005ab0 tx=0x7f82d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.030+0000 7f82caffd640 1 -- 172.21.15.78:0/3469350385 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f82c409bf20 con 0x7f82d41018d0 2024-03-16T11:05:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.206+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f82d41a0300 con 0x7f82d41018d0 2024-03-16T11:05:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.206+0000 7f82caffd640 1 -- 172.21.15.78:0/3469350385 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f82c409b820 con 0x7f82d41018d0 2024-03-16T11:05:23.207 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82c00a2890 msgr2=0x7f82c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82c00a2890 0x7f82c00a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f82d0005ab0 tx=0x7f82d0005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41018d0 msgr2=0x7f82d4197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41018d0 0x7f82d4197fe0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f82c400e9f0 tx=0x7f82c400edf0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 shutdown_connections 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82d4102290 0x7f82d419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d41018d0 0x7f82d4197fe0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f82c00a2890 0x7f82c00a4d50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 --2- 172.21.15.78:0/3469350385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d41006e0 0x7f82d4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 >> 172.21.15.78:0/3469350385 conn(0x7f82d40fbc70 msgr2=0x7f82d4068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:23.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 shutdown_connections 2024-03-16T11:05:23.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.208+0000 7f82dbe75640 1 -- 172.21.15.78:0/3469350385 wait complete. 2024-03-16T11:05:23.220 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:05:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 -- 172.21.15.78:0/2836820212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 msgr2=0x7fdf1c101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 --2- 172.21.15.78:0/2836820212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c101d60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009900 tx=0x7fdf1002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 -- 172.21.15.78:0/2836820212 shutdown_connections 2024-03-16T11:05:23.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 --2- 172.21.15.78:0/2836820212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1022a0 0x7fdf1c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 --2- 172.21.15.78:0/2836820212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c101d60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 --2- 172.21.15.78:0/2836820212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf1c1006f0 0x7fdf1c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 -- 172.21.15.78:0/2836820212 >> 172.21.15.78:0/2836820212 conn(0x7fdf1c0fbc80 msgr2=0x7fdf1c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 -- 172.21.15.78:0/2836820212 shutdown_connections 2024-03-16T11:05:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.323+0000 7fdf214e8640 1 -- 172.21.15.78:0/2836820212 wait complete. 2024-03-16T11:05:23.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 Processor -- start 2024-03-16T11:05:23.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 -- start start 2024-03-16T11:05:23.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf1c1006f0 0x7fdf1c06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1022a0 0x7fdf1c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c10c220 con 0x7fdf1c1018e0 2024-03-16T11:05:23.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c10c0a0 con 0x7fdf1c1022a0 2024-03-16T11:05:23.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf214e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c10c3a0 con 0x7fdf1c1006f0 2024-03-16T11:05:23.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42814/0 (socket says 172.21.15.78:42814) 2024-03-16T11:05:23.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1a7fc640 1 -- 172.21.15.78:0/4216718645 learned_addr learned my addr 172.21.15.78:0/4216718645 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:23.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1affd640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf1c1006f0 0x7fdf1c06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1b7fe640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1022a0 0x7fdf1c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1a7fc640 1 -- 172.21.15.78:0/4216718645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf1c1006f0 msgr2=0x7fdf1c06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1a7fc640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf1c1006f0 0x7fdf1c06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1a7fc640 1 -- 172.21.15.78:0/4216718645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1022a0 msgr2=0x7fdf1c19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.324+0000 7fdf1a7fc640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1022a0 0x7fdf1c19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.325+0000 7fdf1a7fc640 1 -- 172.21.15.78:0/4216718645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf1c19f6e0 con 0x7fdf1c1018e0 2024-03-16T11:05:23.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.325+0000 7fdf1a7fc640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c19c7e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006010 tx=0x7fdf10004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.325+0000 7fdf03fff640 1 -- 172.21.15.78:0/4216718645 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10043020 con 0x7fdf1c1018e0 2024-03-16T11:05:23.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.325+0000 7fdf03fff640 1 -- 172.21.15.78:0/4216718645 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf10004440 con 0x7fdf1c1018e0 2024-03-16T11:05:23.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.325+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf1c19f970 con 0x7fdf1c1018e0 2024-03-16T11:05:23.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.325+0000 7fdf03fff640 1 -- 172.21.15.78:0/4216718645 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10003bf0 con 0x7fdf1c1018e0 2024-03-16T11:05:23.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.325+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf1c19fec0 con 0x7fdf1c1018e0 2024-03-16T11:05:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.326+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf1c107b20 con 0x7fdf1c1018e0 2024-03-16T11:05:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.327+0000 7fdf03fff640 1 -- 172.21.15.78:0/4216718645 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fdf10046020 con 0x7fdf1c1018e0 2024-03-16T11:05:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.327+0000 7fdf03fff640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdf040a2890 0x7fdf040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.327+0000 7fdf1affd640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdf040a2890 0x7fdf040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.327+0000 7fdf03fff640 1 -- 172.21.15.78:0/4216718645 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdf100f0de0 con 0x7fdf1c1018e0 2024-03-16T11:05:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.328+0000 7fdf1affd640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdf040a2890 0x7fdf040a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c101740 tx=0x7fdf0c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.332+0000 7fdf03fff640 1 -- 172.21.15.78:0/4216718645 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf1004d020 con 0x7fdf1c1018e0 2024-03-16T11:05:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:23 smithi098 ceph-mon[35848]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:23 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/124184867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:23 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/862389513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:23 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3912440514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:23.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.506+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdf1c02d3d0 con 0x7fdf1c1018e0 2024-03-16T11:05:23.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.506+0000 7fdf03fff640 1 -- 172.21.15.78:0/4216718645 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fdf100bde30 con 0x7fdf1c1018e0 2024-03-16T11:05:23.507 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdf040a2890 msgr2=0x7fdf040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdf040a2890 0x7fdf040a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c101740 tx=0x7fdf0c009500 comp rx=0 tx=0).stop 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 msgr2=0x7fdf1c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c19c7e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006010 tx=0x7fdf10004040 comp rx=0 tx=0).stop 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 shutdown_connections 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1022a0 0x7fdf1c19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf1c1018e0 0x7fdf1c19c7e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdf040a2890 0x7fdf040a4d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 --2- 172.21.15.78:0/4216718645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf1c1006f0 0x7fdf1c06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 >> 172.21.15.78:0/4216718645 conn(0x7fdf1c0fbc80 msgr2=0x7fdf1c104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 shutdown_connections 2024-03-16T11:05:23.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.508+0000 7fdf214e8640 1 -- 172.21.15.78:0/4216718645 wait complete. 2024-03-16T11:05:23.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:23 smithi078 ceph-mon[32235]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:23.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:23 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/124184867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:23.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:23 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/862389513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:23.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:23 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3912440514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:23.520 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:05:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:23 smithi153 ceph-mon[35521]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:23 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/124184867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:23 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/862389513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:23 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3912440514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:23.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 -- 172.21.15.78:0/3943009576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101370 msgr2=0x7f836c108c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 --2- 172.21.15.78:0/3943009576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101370 0x7f836c108c10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8368009930 tx=0x7f836802bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 -- 172.21.15.78:0/3943009576 shutdown_connections 2024-03-16T11:05:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 --2- 172.21.15.78:0/3943009576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101370 0x7f836c108c10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 --2- 172.21.15.78:0/3943009576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c1009b0 0x7f836c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 --2- 172.21.15.78:0/3943009576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f836c101d10 0x7f836c102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 -- 172.21.15.78:0/3943009576 >> 172.21.15.78:0/3943009576 conn(0x7f836c0fb400 msgr2=0x7f836c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 -- 172.21.15.78:0/3943009576 shutdown_connections 2024-03-16T11:05:23.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 -- 172.21.15.78:0/3943009576 wait complete. 2024-03-16T11:05:23.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 Processor -- start 2024-03-16T11:05:23.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.621+0000 7f83740f8640 1 -- start start 2024-03-16T11:05:23.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f836c1009b0 0x7f836c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c101370 0x7f836c196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101d10 0x7f836c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f836c10c260 con 0x7f836c101d10 2024-03-16T11:05:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f836c10c0e0 con 0x7f836c1009b0 2024-03-16T11:05:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f836c10c3e0 con 0x7f836c101370 2024-03-16T11:05:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101d10 0x7f836c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101d10 0x7f836c19dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42824/0 (socket says 172.21.15.78:42824) 2024-03-16T11:05:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 -- 172.21.15.78:0/1771170197 learned_addr learned my addr 172.21.15.78:0/1771170197 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8371e94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f836c1009b0 0x7f836c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8371693640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c101370 0x7f836c196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 -- 172.21.15.78:0/1771170197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c101370 msgr2=0x7f836c196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c101370 0x7f836c196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 -- 172.21.15.78:0/1771170197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f836c1009b0 msgr2=0x7f836c196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f836c1009b0 0x7f836c196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 -- 172.21.15.78:0/1771170197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836c19e530 con 0x7f836c101d10 2024-03-16T11:05:23.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8371e94640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f836c1009b0 0x7f836c196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:23.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8372695640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101d10 0x7f836c19dec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8368009900 tx=0x7f8368005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8362ffd640 1 -- 172.21.15.78:0/1771170197 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368004e00 con 0x7f836c101d10 2024-03-16T11:05:23.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8362ffd640 1 -- 172.21.15.78:0/1771170197 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8368005450 con 0x7f836c101d10 2024-03-16T11:05:23.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f836c0ff8a0 con 0x7f836c101d10 2024-03-16T11:05:23.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f8362ffd640 1 -- 172.21.15.78:0/1771170197 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368047480 con 0x7f836c101d10 2024-03-16T11:05:23.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.622+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f836c0ffe40 con 0x7f836c101d10 2024-03-16T11:05:23.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.623+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f836c068a20 con 0x7f836c101d10 2024-03-16T11:05:23.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.624+0000 7f8362ffd640 1 -- 172.21.15.78:0/1771170197 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8368004fa0 con 0x7f836c101d10 2024-03-16T11:05:23.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.625+0000 7f8362ffd640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83580a2910 0x7f83580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.625+0000 7f8371e94640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83580a2910 0x7f83580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.625+0000 7f8362ffd640 1 -- 172.21.15.78:0/1771170197 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f83680f2c80 con 0x7f836c101d10 2024-03-16T11:05:23.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.625+0000 7f8371e94640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83580a2910 0x7f83580a4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f83640099a0 tx=0x7f8364009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.629+0000 7f8362ffd640 1 -- 172.21.15.78:0/1771170197 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f83680bfcd0 con 0x7f836c101d10 2024-03-16T11:05:23.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.841+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f836c107710 con 0x7f836c101d10 2024-03-16T11:05:23.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.842+0000 7f8362ffd640 1 -- 172.21.15.78:0/1771170197 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11869 (secure 0 0 0) 0x7f83680bf5d0 con 0x7f836c101d10 2024-03-16T11:05:23.842 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:23.842 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:04:03.658334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24488,24505,24521]},"id":1}]} 2024-03-16T11:05:23.843 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 21 2024-03-16T11:05:23.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83580a2910 msgr2=0x7f83580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83580a2910 0x7f83580a4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f83640099a0 tx=0x7f8364009500 comp rx=0 tx=0).stop 2024-03-16T11:05:23.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101d10 msgr2=0x7f836c19dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101d10 0x7f836c19dec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8368009900 tx=0x7f8368005e40 comp rx=0 tx=0).stop 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 shutdown_connections 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836c101d10 0x7f836c19dec0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f83580a2910 0x7f83580a4dd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c101370 0x7f836c196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 --2- 172.21.15.78:0/1771170197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f836c1009b0 0x7f836c196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 >> 172.21.15.78:0/1771170197 conn(0x7f836c0fb400 msgr2=0x7f836c104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 shutdown_connections 2024-03-16T11:05:23.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.844+0000 7f83740f8640 1 -- 172.21.15.78:0/1771170197 wait complete. 2024-03-16T11:05:23.856 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 -- 172.21.15.78:0/975733468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 msgr2=0x7faf1c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 --2- 172.21.15.78:0/975733468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 0x7faf1c101df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf04009860 tx=0x7faf0402bb70 comp rx=0 tx=0).stop 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 -- 172.21.15.78:0/975733468 shutdown_connections 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 --2- 172.21.15.78:0/975733468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 0x7faf1c108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 --2- 172.21.15.78:0/975733468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 0x7faf1c101df0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 --2- 172.21.15.78:0/975733468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c100780 0x7faf1c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 -- 172.21.15.78:0/975733468 >> 172.21.15.78:0/975733468 conn(0x7faf1c0fbd10 msgr2=0x7faf1c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 -- 172.21.15.78:0/975733468 shutdown_connections 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.957+0000 7faf21a30640 1 -- 172.21.15.78:0/975733468 wait complete. 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 Processor -- start 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 -- start start 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c100780 0x7faf1c06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 0x7faf1c19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 0x7faf1c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c10c240 con 0x7faf1c102330 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c10c0c0 con 0x7faf1c101970 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf21a30640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c10c3c0 con 0x7faf1c100780 2024-03-16T11:05:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 0x7faf1c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 0x7faf1c19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42838/0 (socket says 172.21.15.78:42838) 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c100780 0x7faf1c06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 -- 172.21.15.78:0/2475959753 learned_addr learned my addr 172.21.15.78:0/2475959753 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf13fff640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 0x7faf1c19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 -- 172.21.15.78:0/2475959753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c100780 msgr2=0x7faf1c06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c100780 0x7faf1c06a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 -- 172.21.15.78:0/2475959753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 msgr2=0x7faf1c19c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 0x7faf1c19c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 -- 172.21.15.78:0/2475959753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c19f6d0 con 0x7faf1c102330 2024-03-16T11:05:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.958+0000 7faf1b7fe640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 0x7faf1c19ee10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf1400efc0 tx=0x7faf1400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.959+0000 7faf20a2e640 1 -- 172.21.15.78:0/2475959753 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14003970 con 0x7faf1c102330 2024-03-16T11:05:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.959+0000 7faf20a2e640 1 -- 172.21.15.78:0/2475959753 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf14004020 con 0x7faf1c102330 2024-03-16T11:05:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.959+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf1c19f9c0 con 0x7faf1c102330 2024-03-16T11:05:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.959+0000 7faf20a2e640 1 -- 172.21.15.78:0/2475959753 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1400b940 con 0x7faf1c102330 2024-03-16T11:05:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.959+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf1c19fe90 con 0x7faf1c102330 2024-03-16T11:05:23.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.960+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf1c107bc0 con 0x7faf1c102330 2024-03-16T11:05:23.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.961+0000 7faf20a2e640 1 -- 172.21.15.78:0/2475959753 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7faf1401d020 con 0x7faf1c102330 2024-03-16T11:05:23.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.961+0000 7faf20a2e640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf080a2890 0x7faf080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:23.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.961+0000 7faf1affd640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf080a2890 0x7faf080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:23.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.961+0000 7faf20a2e640 1 -- 172.21.15.78:0/2475959753 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faf140cea60 con 0x7faf1c102330 2024-03-16T11:05:23.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.962+0000 7faf1affd640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf080a2890 0x7faf080a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faf1c1017d0 tx=0x7faf0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:23.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:23.965+0000 7faf20a2e640 1 -- 172.21.15.78:0/2475959753 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf1409b9a0 con 0x7faf1c102330 2024-03-16T11:05:24.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.142+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf1c02d390 con 0x7faf1c102330 2024-03-16T11:05:24.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.143+0000 7faf20a2e640 1 -- 172.21.15.78:0/2475959753 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7faf1409b2a0 con 0x7faf1c102330 2024-03-16T11:05:24.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:24.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf080a2890 msgr2=0x7faf080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf080a2890 0x7faf080a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faf1c1017d0 tx=0x7faf0c005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:24.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 msgr2=0x7faf1c19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 0x7faf1c19ee10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf1400efc0 tx=0x7faf1400c800 comp rx=0 tx=0).stop 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 shutdown_connections 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf1c102330 0x7faf1c19ee10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf1c101970 0x7faf1c19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf080a2890 0x7faf080a4d50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 --2- 172.21.15.78:0/2475959753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c100780 0x7faf1c06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 >> 172.21.15.78:0/2475959753 conn(0x7faf1c0fbd10 msgr2=0x7faf1c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 shutdown_connections 2024-03-16T11:05:24.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.146+0000 7faf21a30640 1 -- 172.21.15.78:0/2475959753 wait complete. 2024-03-16T11:05:24.158 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:05:24.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 -- 172.21.15.78:0/754363469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 msgr2=0x7fde34102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 --2- 172.21.15.78:0/754363469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34102e80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009900 tx=0x7fde1c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 -- 172.21.15.78:0/754363469 shutdown_connections 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 --2- 172.21.15.78:0/754363469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde341033c0 0x7fde34109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 --2- 172.21.15.78:0/754363469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34102e80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 --2- 172.21.15.78:0/754363469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde34101810 0x7fde34101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 -- 172.21.15.78:0/754363469 >> 172.21.15.78:0/754363469 conn(0x7fde340fcd80 msgr2=0x7fde340ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 -- 172.21.15.78:0/754363469 shutdown_connections 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 -- 172.21.15.78:0/754363469 wait complete. 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 Processor -- start 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.260+0000 7fde3a137640 1 -- start start 2024-03-16T11:05:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34101810 0x7fde34075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde341033c0 0x7fde34074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde3410d340 con 0x7fde34102a00 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde3410d1c0 con 0x7fde34101810 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde3410d4c0 con 0x7fde341033c0 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42858/0 (socket says 172.21.15.78:42858) 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 -- 172.21.15.78:0/1396101285 learned_addr learned my addr 172.21.15.78:0/1396101285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde337fe640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34101810 0x7fde34075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde33fff640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde341033c0 0x7fde34074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 -- 172.21.15.78:0/1396101285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde341033c0 msgr2=0x7fde34074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde341033c0 0x7fde34074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 -- 172.21.15.78:0/1396101285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34101810 msgr2=0x7fde34075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34101810 0x7fde34075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 -- 172.21.15.78:0/1396101285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde340749a0 con 0x7fde34102a00 2024-03-16T11:05:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde32ffd640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34073bd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fde1c006010 tx=0x7fde1c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde30ff9640 1 -- 172.21.15.78:0/1396101285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c043020 con 0x7fde34102a00 2024-03-16T11:05:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde30ff9640 1 -- 172.21.15.78:0/1396101285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde1c004440 con 0x7fde34102a00 2024-03-16T11:05:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde341a1b60 con 0x7fde34102a00 2024-03-16T11:05:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde30ff9640 1 -- 172.21.15.78:0/1396101285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c003bf0 con 0x7fde34102a00 2024-03-16T11:05:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.261+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde341a2100 con 0x7fde34102a00 2024-03-16T11:05:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.262+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde34108c00 con 0x7fde34102a00 2024-03-16T11:05:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.263+0000 7fde30ff9640 1 -- 172.21.15.78:0/1396101285 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fde1c046020 con 0x7fde34102a00 2024-03-16T11:05:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.264+0000 7fde30ff9640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fde200a2910 0x7fde200a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.264+0000 7fde337fe640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fde200a2910 0x7fde200a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.264+0000 7fde30ff9640 1 -- 172.21.15.78:0/1396101285 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fde1c0f0d60 con 0x7fde34102a00 2024-03-16T11:05:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.264+0000 7fde337fe640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fde200a2910 0x7fde200a4dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fde34102860 tx=0x7fde24009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.268+0000 7fde30ff9640 1 -- 172.21.15.78:0/1396101285 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde1c04d020 con 0x7fde34102a00 2024-03-16T11:05:24.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:24 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3469350385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:24.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:24 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4216718645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:24.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:24 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1771170197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:24.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:24 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2475959753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:24.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.483+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde3402d1b0 con 0x7fde34102a00 2024-03-16T11:05:24.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.484+0000 7fde30ff9640 1 -- 172.21.15.78:0/1396101285 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11869 (secure 0 0 0) 0x7fde1c0dd070 con 0x7fde34102a00 2024-03-16T11:05:24.484 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:24.484 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:04:03.658334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24488,24505,24521]},"id":1}]} 2024-03-16T11:05:24.485 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 21 2024-03-16T11:05:24.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fde200a2910 msgr2=0x7fde200a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fde200a2910 0x7fde200a4dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fde34102860 tx=0x7fde24009680 comp rx=0 tx=0).stop 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 msgr2=0x7fde34073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34073bd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fde1c006010 tx=0x7fde1c004040 comp rx=0 tx=0).stop 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 shutdown_connections 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fde200a2910 0x7fde200a4dd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde341033c0 0x7fde34074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde34102a00 0x7fde34073bd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 --2- 172.21.15.78:0/1396101285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34101810 0x7fde34075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 >> 172.21.15.78:0/1396101285 conn(0x7fde340fcd80 msgr2=0x7fde34105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 shutdown_connections 2024-03-16T11:05:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.486+0000 7fde3a137640 1 -- 172.21.15.78:0/1396101285 wait complete. 2024-03-16T11:05:24.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:24 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3469350385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:24.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:24 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4216718645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:24.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:24 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1771170197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:24.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:24 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2475959753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:24.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T11:03:59.617240+0000', 'modified': '2024-03-16T11:04:03.658334+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24505, 'mds_2': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24458': {'gid': 24458, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6835/1190197046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1190197046}, {'type': 'v1', 'addr': '172.21.15.98:6835', 'nonce': 1190197046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/1109965992', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 1109965992}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 1109965992}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/3505861862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 3505861862}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 3505861862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.98:6837/1752557490', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6836', 'nonce': 1752557490}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 1752557490}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.153:6839/1504943793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 1504943793}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 1504943793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6839/4092843053', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 4092843053}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 4092843053}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24488, 24505, 24521]} 2024-03-16T11:05:24.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T11:05:24.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T11:05:24.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:24 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3469350385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:24.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:24 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4216718645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:24.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:24 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1771170197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:24.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:24 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2475959753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:24.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 -- 172.21.15.78:0/3719826167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681018e0 msgr2=0x7f4168101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 --2- 172.21.15.78:0/3719826167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681018e0 0x7f4168101d60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f415c009900 tx=0x7f415c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 -- 172.21.15.78:0/3719826167 shutdown_connections 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 --2- 172.21.15.78:0/3719826167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41681022a0 0x7f4168108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 --2- 172.21.15.78:0/3719826167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681018e0 0x7f4168101d60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 --2- 172.21.15.78:0/3719826167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41681006f0 0x7f4168100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 -- 172.21.15.78:0/3719826167 >> 172.21.15.78:0/3719826167 conn(0x7f41680fbc80 msgr2=0x7f41680fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 -- 172.21.15.78:0/3719826167 shutdown_connections 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.604+0000 7f416dcbf640 1 -- 172.21.15.78:0/3719826167 wait complete. 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 Processor -- start 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 -- start start 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681006f0 0x7f4168197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41681018e0 0x7f4168197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41681022a0 0x7f416819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f416810d340 con 0x7f41681006f0 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f416810d1c0 con 0x7f41681018e0 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f416dcbf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f416810d4c0 con 0x7f41681022a0 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681006f0 0x7f4168197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681006f0 0x7f4168197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42876/0 (socket says 172.21.15.78:42876) 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 -- 172.21.15.78:0/2392064718 learned_addr learned my addr 172.21.15.78:0/2392064718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f4167fff640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41681022a0 0x7f416819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f4166ffd640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41681018e0 0x7f4168197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 -- 172.21.15.78:0/2392064718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41681022a0 msgr2=0x7f416819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41681022a0 0x7f416819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 -- 172.21.15.78:0/2392064718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41681018e0 msgr2=0x7f4168197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41681018e0 0x7f4168197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 -- 172.21.15.78:0/2392064718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416819f630 con 0x7f41681006f0 2024-03-16T11:05:24.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.605+0000 7f41677fe640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681006f0 0x7f4168197a10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f415800e9c0 tx=0x7f415800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.606+0000 7f4164ff9640 1 -- 172.21.15.78:0/2392064718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415800cde0 con 0x7f41681006f0 2024-03-16T11:05:24.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.606+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f416819f920 con 0x7f41681006f0 2024-03-16T11:05:24.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.606+0000 7f4164ff9640 1 -- 172.21.15.78:0/2392064718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41580108e0 con 0x7f41681006f0 2024-03-16T11:05:24.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.606+0000 7f4164ff9640 1 -- 172.21.15.78:0/2392064718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580188e0 con 0x7f41681006f0 2024-03-16T11:05:24.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.606+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f416819fe90 con 0x7f41681006f0 2024-03-16T11:05:24.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.608+0000 7f4164ff9640 1 -- 172.21.15.78:0/2392064718 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f415801f020 con 0x7f41681006f0 2024-03-16T11:05:24.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.608+0000 7f4170981740 1 Processor -- start 2024-03-16T11:05:24.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.608+0000 7f4170981740 1 -- start start 2024-03-16T11:05:24.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4170981740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x56183181cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4170981740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x5618318220c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4170981740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831824bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4170981740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561831780c80 con 0x56183181d5c0 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4170981740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561831729aa0 con 0x56183181cbd0 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4170981740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56183177cd20 con 0x561831822730 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x5618318220c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4164ff9640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f41500a2890 0x7f41500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4167fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x56183181cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x5618318220c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42882/0 (socket says 172.21.15.78:42882) 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 -- 172.21.15.78:0/494413608 learned_addr learned my addr 172.21.15.78:0/494413608 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4167fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x56183181cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:34582/0 (socket says 172.21.15.78:34582) 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4166ffd640 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831824bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4167fff640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f41500a2890 0x7f41500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4164ff9640 1 -- 172.21.15.78:0/2392064718 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f41580cedb0 con 0x7f41681006f0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 -- 172.21.15.78:0/494413608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 msgr2=0x561831824bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831824bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 -- 172.21.15.78:0/494413608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 msgr2=0x56183181cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x56183181cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 -- 172.21.15.78:0/494413608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56183174bff0 con 0x56183181d5c0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4166ffd640 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831824bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4167fff640 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x56183181cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f41677fe640 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x5618318220c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4158097a50 tx=0x7f415800efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee67560f4aec95a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f4167fff640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f41500a2890 0x7f41500a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x56183181de20 tx=0x7f4160004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.609+0000 7f414dffb640 1 -- 172.21.15.78:0/494413608 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158097030 con 0x56183181d5c0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f414dffb640 1 -- 172.21.15.78:0/494413608 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f415809b5f0 con 0x56183181d5c0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f414dffb640 1 -- 172.21.15.78:0/494413608 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415809b910 con 0x56183181d5c0 2024-03-16T11:05:24.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 -- 172.21.15.78:0/494413608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 msgr2=0x5618318220c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x5618318220c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4158097a50 tx=0x7f415800efb0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 -- 172.21.15.78:0/494413608 shutdown_connections 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831824bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x5618318220c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 --2- 172.21.15.78:0/494413608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x56183181cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 -- 172.21.15.78:0/494413608 >> 172.21.15.78:0/494413608 conn(0x561831818140 msgr2=0x561831818540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 -- 172.21.15.78:0/494413608 shutdown_connections 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 -- 172.21.15.78:0/494413608 wait complete. 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.610+0000 7f4170981740 1 -- start start 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 1 Processor -- start 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831826d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831827290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x56183182e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561831825620 con 0x56183181d5c0 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56183173cef0 con 0x56183181cbd0 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5618318257a0 con 0x561831822730 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4170981740 10 client.0 _ll_register_callbacks cb 0x561831737dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831827290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831827290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42892/0 (socket says 172.21.15.78:42892) 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f41677fe640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831826d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 -- 172.21.15.78:0/3752898763 learned_addr learned my addr 172.21.15.78:0/3752898763 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4167fff640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x56183182e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 -- 172.21.15.78:0/3752898763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 msgr2=0x56183182e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x56183182e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 -- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 msgr2=0x561831826d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831826d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f4166ffd640 1 -- 172.21.15.78:0/3752898763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561831749a30 con 0x56183181d5c0 2024-03-16T11:05:24.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.611+0000 7f41677fe640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831826d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4166ffd640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831827290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f415c02c170 tx=0x7f415c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7bbfdc06f343883 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4146ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4146ffd640 1 -- 172.21.15.78:0/3752898763 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c005d50 con 0x56183181d5c0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4146ffd640 1 -- 172.21.15.78:0/3752898763 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f415c005030 con 0x56183181d5c0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56183178e1c0 con 0x56183181d5c0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4146ffd640 1 -- 172.21.15.78:0/3752898763 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f415c005230 con 0x56183181d5c0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 10 client.15075 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5618314952d0 con 0x56183181d5c0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4146ffd640 1 -- 172.21.15.78:0/3752898763 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56183178e1c0 con 0x56183181d5c0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4146ffd640 1 -- 172.21.15.78:0/3752898763 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f415c003680 con 0x56183181d5c0 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 10 client.15075 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 10 client.15075 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 20 client.15075 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x561831834ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 4 client.15075 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561831788510 con 0x561831833830 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.612+0000 7f41677fe640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x561831834ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.613+0000 7f4146ffd640 10 client.15075 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.613+0000 7f41677fe640 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x561831834ca0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd2f2b2292b8d47d server_cookie=6b5242451db22d46 in_seq=0 out_seq=0 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.613+0000 7f4146ffd640 1 -- 172.21.15.78:0/3752898763 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561831788510 con 0x561831833830 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.613+0000 7f4146ffd640 10 client.15075 handle_command_reply: tid=0 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.613+0000 7f4170981740 1 client.15075 shutdown 2024-03-16T11:05:24.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 msgr2=0x561831827290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831827290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f415c02c170 tx=0x7f415c004270 comp rx=0 tx=0).stop 2024-03-16T11:05:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 shutdown_connections 2024-03-16T11:05:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x561831834ca0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x56183182e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4146ffd640 0 client.15075 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831827290 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 --2- 172.21.15.78:0/3752898763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831826d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 >> 172.21.15.78:0/3752898763 conn(0x561831818140 msgr2=0x561831817380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 shutdown_connections 2024-03-16T11:05:24.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 1 -- 172.21.15.78:0/3752898763 wait complete. 2024-03-16T11:05:24.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.614+0000 7f4170981740 20 client.15075 trim_cache size 0 max 16384 2024-03-16T11:05:24.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 Processor -- start 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 -- start start 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56183181cbd0 0x561831821da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x56183182e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561831822730 0x561831826960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561831832fe0 con 0x56183181d5c0 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5618318336b0 con 0x561831822730 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4170981740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561831833d80 con 0x56183181cbd0 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x56183182e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x56183182e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42896/0 (socket says 172.21.15.78:42896) 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 -- 172.21.15.78:0/3158402991 learned_addr learned my addr 172.21.15.78:0/3158402991 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f41677fe640 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56183181cbd0 0x561831821da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4167fff640 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561831822730 0x561831826960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 -- 172.21.15.78:0/3158402991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56183181cbd0 msgr2=0x561831821da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56183181cbd0 0x561831821da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 -- 172.21.15.78:0/3158402991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561831822730 msgr2=0x561831826960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561831822730 0x561831826960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 -- 172.21.15.78:0/3158402991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56183155f6b0 con 0x56183181d5c0 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f41677fe640 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56183181cbd0 0x561831821da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4167fff640 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561831822730 0x561831826960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f4166ffd640 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x56183182e290 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f415c009850 tx=0x7f415c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab4efdaef0e2931 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.632+0000 7f414d7fa640 1 -- 172.21.15.78:0/3158402991 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c005d50 con 0x56183181d5c0 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f414d7fa640 1 -- 172.21.15.78:0/3158402991 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f415c009520 con 0x56183181d5c0 2024-03-16T11:05:24.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f414d7fa640 1 -- 172.21.15.78:0/3158402991 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c040a10 con 0x56183181d5c0 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 -- 172.21.15.78:0/3158402991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 msgr2=0x56183182e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x56183182e290 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f415c009850 tx=0x7f415c0058d0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 -- 172.21.15.78:0/3158402991 shutdown_connections 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561831822730 0x561831826960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x56183182e290 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 --2- 172.21.15.78:0/3158402991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56183181cbd0 0x561831821da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 -- 172.21.15.78:0/3158402991 >> 172.21.15.78:0/3158402991 conn(0x561831818140 msgr2=0x561831818520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 -- 172.21.15.78:0/3158402991 shutdown_connections 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 -- 172.21.15.78:0/3158402991 wait complete. 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.633+0000 7f4170981740 1 -- start start 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 Processor -- start 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831829050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831829590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831897590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561831834b20 con 0x56183181d5c0 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561831834450 con 0x56183181cbd0 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5618318351f0 con 0x561831822730 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 10 client.0 _ll_register_callbacks cb 0x7f415c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831829590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f41677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831829050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831829590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42898/0 (socket says 172.21.15.78:42898) 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 -- 172.21.15.78:0/172353620 learned_addr learned my addr 172.21.15.78:0/172353620 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4167fff640 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831897590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 -- 172.21.15.78:0/172353620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 msgr2=0x561831897590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831897590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 -- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 msgr2=0x561831829050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831829050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 -- 172.21.15.78:0/172353620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561831829c00 con 0x56183181d5c0 2024-03-16T11:05:24.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f41677fe640 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831829050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4166ffd640 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831829590 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f415c004650 tx=0x7f415c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5247ba80fd2d4749 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4146ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4146ffd640 1 -- 172.21.15.78:0/172353620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c063060 con 0x56183181d5c0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4146ffd640 1 -- 172.21.15.78:0/172353620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f415c040790 con 0x56183181d5c0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.634+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f415c009520 con 0x56183181d5c0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4146ffd640 1 -- 172.21.15.78:0/172353620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f415c040990 con 0x56183181d5c0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 10 client.15081 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561831898bd0 con 0x56183181d5c0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4146ffd640 1 -- 172.21.15.78:0/172353620 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f415c009520 con 0x56183181d5c0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4146ffd640 1 -- 172.21.15.78:0/172353620 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f415c068070 con 0x56183181d5c0 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 10 client.15081 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 10 client.15081 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 20 client.15081 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:24.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x56183189c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 4 client.15081 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "status"}) -- 0x561831706a00 con 0x561831833830 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.635+0000 7f41677fe640 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x56183189c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4146ffd640 10 client.15081 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f41677fe640 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x56183189c190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8af7f0f08bf9111c server_cookie=7f766ad570e7ed34 in_seq=0 out_seq=0 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4146ffd640 1 -- 172.21.15.78:0/172353620 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561831706a00 con 0x561831833830 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4146ffd640 10 client.15081 handle_command_reply: tid=0 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 client.15081 shutdown 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 msgr2=0x561831829590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831829590 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f415c004650 tx=0x7f415c0038a0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 shutdown_connections 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x561831833830 0x56183189c190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561831822730 0x561831897590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4146ffd640 0 client.15081 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56183181d5c0 0x561831829590 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 --2- 172.21.15.78:0/172353620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56183181cbd0 0x561831829050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 >> 172.21.15.78:0/172353620 conn(0x561831818140 msgr2=0x561831832610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 shutdown_connections 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 1 -- 172.21.15.78:0/172353620 wait complete. 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.636+0000 7f4170981740 20 client.15081 trim_cache size 0 max 16384 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-03-16T11:05:24.647 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24488, 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 81.331916540999998, 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 21, 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 90, 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 88, 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 100.760919386 2024-03-16T11:05:24.648 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:05:24.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f41500a2890 msgr2=0x7f41500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f41500a2890 0x7f41500a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x56183181de20 tx=0x7f4160004ab0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681006f0 msgr2=0x7f4168197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681006f0 0x7f4168197a10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f415800e9c0 tx=0x7f415800ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 shutdown_connections 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f41500a2890 0x7f41500a4d50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41681022a0 0x7f416819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41681018e0 0x7f4168197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 --2- 172.21.15.78:0/2392064718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41681006f0 0x7f4168197a10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 >> 172.21.15.78:0/2392064718 conn(0x7f41680fbc80 msgr2=0x7f4168104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 shutdown_connections 2024-03-16T11:05:24.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.637+0000 7f416dcbf640 1 -- 172.21.15.78:0/2392064718 wait complete. 2024-03-16T11:05:24.650 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 -- 172.21.15.78:0/2605255418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc102a90 msgr2=0x7f91fc102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 --2- 172.21.15.78:0/2605255418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc102a90 0x7f91fc102f10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009930 tx=0x7f91f002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 -- 172.21.15.78:0/2605255418 shutdown_connections 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 --2- 172.21.15.78:0/2605255418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91fc103450 0x7f91fc109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 --2- 172.21.15.78:0/2605255418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc102a90 0x7f91fc102f10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 --2- 172.21.15.78:0/2605255418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91fc0ffac0 0x7f91fc0ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 -- 172.21.15.78:0/2605255418 >> 172.21.15.78:0/2605255418 conn(0x7f91fc0fb3f0 msgr2=0x7f91fc0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 -- 172.21.15.78:0/2605255418 shutdown_connections 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.752+0000 7f9202073640 1 -- 172.21.15.78:0/2605255418 wait complete. 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 Processor -- start 2024-03-16T11:05:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 -- start start 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91fc0ffac0 0x7f91fc102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91fc102a90 0x7f91fc100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc103450 0x7f91fc100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f91fc10d330 con 0x7f91fc103450 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91fc10d1b0 con 0x7f91fc0ffac0 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f9202073640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f91fc10d4b0 con 0x7f91fc102a90 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc103450 0x7f91fc100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc103450 0x7f91fc100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42922/0 (socket says 172.21.15.78:42922) 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 -- 172.21.15.78:0/2162591900 learned_addr learned my addr 172.21.15.78:0/2162591900 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91fc0ffac0 0x7f91fc102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91faffd640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91fc102a90 0x7f91fc100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 -- 172.21.15.78:0/2162591900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91fc102a90 msgr2=0x7f91fc100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91fc102a90 0x7f91fc100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 -- 172.21.15.78:0/2162591900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91fc0ffac0 msgr2=0x7f91fc102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91fc0ffac0 0x7f91fc102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.753+0000 7f91fbfff640 1 -- 172.21.15.78:0/2162591900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91fc101810 con 0x7f91fc103450 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.754+0000 7f91fbfff640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc103450 0x7f91fc100f80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f91f400eaa0 tx=0x7f91f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.754+0000 7f91f8ff9640 1 -- 172.21.15.78:0/2162591900 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f400cdb0 con 0x7f91fc103450 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.754+0000 7f91f8ff9640 1 -- 172.21.15.78:0/2162591900 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91f4005020 con 0x7f91fc103450 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.754+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91fc1a1b80 con 0x7f91fc103450 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.754+0000 7f91f8ff9640 1 -- 172.21.15.78:0/2162591900 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f40116d0 con 0x7f91fc103450 2024-03-16T11:05:24.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.754+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91fc1a2050 con 0x7f91fc103450 2024-03-16T11:05:24.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.756+0000 7f91f8ff9640 1 -- 172.21.15.78:0/2162591900 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f91f40051d0 con 0x7f91fc103450 2024-03-16T11:05:24.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.756+0000 7f9204c5e740 1 Processor -- start 2024-03-16T11:05:24.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.756+0000 7f9204c5e740 1 -- start start 2024-03-16T11:05:24.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.756+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.756+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x5576215a80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.756+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a8730 0x5576215aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.756+0000 7f9204c5e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557621506c80 con 0x5576215a35c0 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f9204c5e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576214afaa0 con 0x5576215a2bd0 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f9204c5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557621502d20 con 0x5576215a8730 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91f8ff9640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f91e40a2890 0x7f91e40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x5576215a80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x5576215a80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42928/0 (socket says 172.21.15.78:42928) 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 -- 172.21.15.78:0/1710820506 learned_addr learned my addr 172.21.15.78:0/1710820506 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91faffd640 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fbfff640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f91e40a2890 0x7f91e40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91faffd640 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a8730 0x5576215aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 -- 172.21.15.78:0/1710820506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a8730 msgr2=0x5576215aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a8730 0x5576215aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91f8ff9640 1 -- 172.21.15.78:0/2162591900 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f91f40ceda0 con 0x7f91fc103450 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 -- 172.21.15.78:0/1710820506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 msgr2=0x5576215a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215a2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 -- 172.21.15.78:0/1710820506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576214d1ff0 con 0x5576215a35c0 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91faffd640 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fbfff640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f91e40a2890 0x7f91e40a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f91f406e350 tx=0x7f91f40c5ff0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x5576215a80c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009ca0 tx=0x7f91ec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6391a3c852885f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91e1ffb640 1 -- 172.21.15.78:0/1710820506 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec004220 con 0x5576215a35c0 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91e1ffb640 1 -- 172.21.15.78:0/1710820506 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91ec0043c0 con 0x5576215a35c0 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.757+0000 7f91e1ffb640 1 -- 172.21.15.78:0/1710820506 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec019830 con 0x5576215a35c0 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 -- 172.21.15.78:0/1710820506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 msgr2=0x5576215a80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x5576215a80c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009ca0 tx=0x7f91ec011810 comp rx=0 tx=0).stop 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 -- 172.21.15.78:0/1710820506 shutdown_connections 2024-03-16T11:05:24.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a8730 0x5576215aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x5576215a80c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1710820506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 -- 172.21.15.78:0/1710820506 >> 172.21.15.78:0/1710820506 conn(0x55762159e140 msgr2=0x55762159e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 -- 172.21.15.78:0/1710820506 shutdown_connections 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 -- 172.21.15.78:0/1710820506 wait complete. 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.758+0000 7f9204c5e740 1 -- start start 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 Processor -- start 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215ad290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a8730 0x5576215b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5576215ab6e0 con 0x5576215a8730 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576214c2ef0 con 0x5576215a2bd0 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5576215ab860 con 0x5576215a35c0 2024-03-16T11:05:24.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 10 client.0 _ll_register_callbacks cb 0x5576214bddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a8730 0x5576215b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a8730 0x5576215b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42936/0 (socket says 172.21.15.78:42936) 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 -- 172.21.15.78:0/2834506509 learned_addr learned my addr 172.21.15.78:0/2834506509 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 -- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 msgr2=0x5576215ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 -- 172.21.15.78:0/2834506509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 msgr2=0x5576215acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 -- 172.21.15.78:0/2834506509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576214cfa30 con 0x5576215a8730 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91faffd640 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a8730 0x5576215b42d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f91f0036040 tx=0x7f91f002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ce606860fdbc09 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91daffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91daffd640 1 -- 172.21.15.78:0/2834506509 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00054f0 con 0x5576215a8730 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f91daffd640 1 -- 172.21.15.78:0/2834506509 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f91f00046c0 con 0x5576215a8730 2024-03-16T11:05:24.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.759+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91ec0043c0 con 0x5576215a8730 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f91daffd640 1 -- 172.21.15.78:0/2834506509 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f91f00048c0 con 0x5576215a8730 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 10 client.15093 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55762121b2d0 con 0x5576215a8730 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f91daffd640 1 -- 172.21.15.78:0/2834506509 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91ec0043c0 con 0x5576215a8730 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f91daffd640 1 -- 172.21.15.78:0/2834506509 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f91f0003b20 con 0x5576215a8730 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 10 client.15093 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 10 client.15093 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 20 client.15093 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5576215b97e0 0x5576215bac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 4 client.15093 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576215141c0 con 0x5576215b97e0 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.760+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5576215b97e0 0x5576215bac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.761+0000 7f91daffd640 10 client.15093 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:24.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.761+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5576215b97e0 0x5576215bac50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3d905921a956ec47 server_cookie=187cc07eea90fc7b in_seq=0 out_seq=0 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f91daffd640 1 -- 172.21.15.78:0/2834506509 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5576215141c0 con 0x5576215b97e0 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f91daffd640 10 client.15093 handle_command_reply: tid=0 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 client.15093 shutdown 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a8730 msgr2=0x5576215b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a8730 0x5576215b42d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f91f0036040 tx=0x7f91f002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 shutdown_connections 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5576215b97e0 0x5576215bac50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a8730 0x5576215b42d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 --2- 172.21.15.78:0/2834506509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a2bd0 0x5576215acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f91daffd640 0 client.15093 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 >> 172.21.15.78:0/2834506509 conn(0x55762159e140 msgr2=0x55762159d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 shutdown_connections 2024-03-16T11:05:24.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 1 -- 172.21.15.78:0/2834506509 wait complete. 2024-03-16T11:05:24.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.762+0000 7f9204c5e740 20 client.15093 trim_cache size 0 max 16384 2024-03-16T11:05:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 Processor -- start 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 -- start start 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a2bd0 0x5576215ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x5576215acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5576215b8f90 con 0x5576215a2bd0 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576215b9660 con 0x5576215a8730 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f9204c5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5576215b9d30 con 0x5576215a35c0 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a2bd0 0x5576215ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a2bd0 0x5576215ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42950/0 (socket says 172.21.15.78:42950) 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 -- 172.21.15.78:0/1423317853 learned_addr learned my addr 172.21.15.78:0/1423317853 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x5576215acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 -- 172.21.15.78:0/1423317853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 msgr2=0x5576215b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 -- 172.21.15.78:0/1423317853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 msgr2=0x5576215acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x5576215acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 -- 172.21.15.78:0/1423317853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576212e56b0 con 0x5576215a2bd0 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.780+0000 7f91faffd640 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a2bd0 0x5576215ac960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f91f00058d0 tx=0x7f91f0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad31b08e7db8b389 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f91e17fa640 1 -- 172.21.15.78:0/1423317853 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00435d0 con 0x5576215a2bd0 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f91e17fa640 1 -- 172.21.15.78:0/1423317853 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91f00054f0 con 0x5576215a2bd0 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f91e17fa640 1 -- 172.21.15.78:0/1423317853 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00035f0 con 0x5576215a2bd0 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 -- 172.21.15.78:0/1423317853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a2bd0 msgr2=0x5576215ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a2bd0 0x5576215ac960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f91f00058d0 tx=0x7f91f0005c90 comp rx=0 tx=0).stop 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 -- 172.21.15.78:0/1423317853 shutdown_connections 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x5576215acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a35c0 0x5576215b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 --2- 172.21.15.78:0/1423317853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a2bd0 0x5576215ac960 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 -- 172.21.15.78:0/1423317853 >> 172.21.15.78:0/1423317853 conn(0x55762159e140 msgr2=0x5576215b85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 -- 172.21.15.78:0/1423317853 shutdown_connections 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 -- 172.21.15.78:0/1423317853 wait complete. 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.781+0000 7f9204c5e740 1 -- start start 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 1 Processor -- start 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a2bd0 0x5576215b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x55762161b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x55762161da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5576215baad0 con 0x5576215a35c0 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576215ba400 con 0x5576215a8730 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5576215bb1a0 con 0x5576215a2bd0 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f9204c5e740 10 client.0 _ll_register_callbacks cb 0x7f91f0004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x55762161b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x55762161b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42958/0 (socket says 172.21.15.78:42958) 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 -- 172.21.15.78:0/3367099985 learned_addr learned my addr 172.21.15.78:0/3367099985 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91faffd640 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a2bd0 0x5576215b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91faffd640 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x55762161da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 -- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a2bd0 msgr2=0x5576215b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a2bd0 0x5576215b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 -- 172.21.15.78:0/3367099985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 msgr2=0x55762161da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x55762161da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 -- 172.21.15.78:0/3367099985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55762161dfc0 con 0x5576215a35c0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91fb7fe640 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x55762161b530 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009ca0 tx=0x7f91ec005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfa50ce37b8dcd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.782+0000 7f91daffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f91daffd640 1 -- 172.21.15.78:0/3367099985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec02bc40 con 0x5576215a35c0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f91daffd640 1 -- 172.21.15.78:0/3367099985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f91ec02bde0 con 0x5576215a35c0 2024-03-16T11:05:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91f00054f0 con 0x5576215a35c0 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f91daffd640 1 -- 172.21.15.78:0/3367099985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f91ec02b320 con 0x5576215a35c0 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 10 client.15099 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55762161f2d0 con 0x5576215a35c0 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f91daffd640 1 -- 172.21.15.78:0/3367099985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91f00054f0 con 0x5576215a35c0 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f91daffd640 1 -- 172.21.15.78:0/3367099985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f91ec0035f0 con 0x5576215a35c0 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 10 client.15099 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 10 client.15099 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 20 client.15099 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x557621623cc0 0x5576216229d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 4 client.15099 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "status"}) -- 0x55762150e510 con 0x557621623cc0 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.783+0000 7f91faffd640 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x557621623cc0 0x5576216229d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f91daffd640 10 client.15099 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f91faffd640 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x557621623cc0 0x5576216229d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=53ba72889c825170 server_cookie=a0682e5aadd86763 in_seq=0 out_seq=0 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f91daffd640 1 -- 172.21.15.78:0/3367099985 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55762150e510 con 0x557621623cc0 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f91daffd640 10 client.15099 handle_command_reply: tid=0 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 client.15099 shutdown 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 msgr2=0x55762161b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x55762161b530 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009ca0 tx=0x7f91ec005d50 comp rx=0 tx=0).stop 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 shutdown_connections 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x557621623cc0 0x5576216229d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576215a8730 0x55762161da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5576215a35c0 0x55762161b530 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f91daffd640 0 client.15099 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 --2- 172.21.15.78:0/3367099985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5576215a2bd0 0x5576215b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 >> 172.21.15.78:0/3367099985 conn(0x55762159e140 msgr2=0x5576215ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 shutdown_connections 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 1 -- 172.21.15.78:0/3367099985 wait complete. 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.784+0000 7f9204c5e740 20 client.15099 trim_cache size 0 max 16384 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24505, 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-16T11:05:24.792 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 82.484989322000004, 2024-03-16T11:05:24.793 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 21, 2024-03-16T11:05:24.793 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 90, 2024-03-16T11:05:24.793 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 88, 2024-03-16T11:05:24.793 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 98.263253156000005 2024-03-16T11:05:24.793 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:05:24.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f91e40a2890 msgr2=0x7f91e40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f91e40a2890 0x7f91e40a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f91f406e350 tx=0x7f91f40c5ff0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc103450 msgr2=0x7f91fc100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc103450 0x7f91fc100f80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f91f400eaa0 tx=0x7f91f400ef60 comp rx=0 tx=0).stop 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 shutdown_connections 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91fc103450 0x7f91fc100f80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f91e40a2890 0x7f91e40a4d50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91fc102a90 0x7f91fc100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 --2- 172.21.15.78:0/2162591900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91fc0ffac0 0x7f91fc102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 >> 172.21.15.78:0/2162591900 conn(0x7f91fc0fb3f0 msgr2=0x7f91fc105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 shutdown_connections 2024-03-16T11:05:24.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.785+0000 7f9202073640 1 -- 172.21.15.78:0/2162591900 wait complete. 2024-03-16T11:05:24.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T11:05:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 -- 172.21.15.78:0/1470108131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017f0 msgr2=0x7fdfd8101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/1470108131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017f0 0x7fdfd8101c10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009be0 tx=0x7fdfc802bef0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 -- 172.21.15.78:0/1470108131 shutdown_connections 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/1470108131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd81033a0 0x7fdfd8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/1470108131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81029e0 0x7fdfd8102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/1470108131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017f0 0x7fdfd8101c10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 -- 172.21.15.78:0/1470108131 >> 172.21.15.78:0/1470108131 conn(0x7fdfd80fcd80 msgr2=0x7fdfd80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 -- 172.21.15.78:0/1470108131 shutdown_connections 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.897+0000 7fdfdd61f640 1 -- 172.21.15.78:0/1470108131 wait complete. 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 Processor -- start 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 -- start start 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd81017f0 0x7fdfd8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81029e0 0x7fdfd8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81033a0 0x7fdfd819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810d320 con 0x7fdfd81033a0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810d1a0 con 0x7fdfd81029e0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfdd61f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810d4a0 con 0x7fdfd81017f0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81033a0 0x7fdfd819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81033a0 0x7fdfd819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42980/0 (socket says 172.21.15.78:42980) 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd81017f0 0x7fdfd8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 -- 172.21.15.78:0/3121225635 learned_addr learned my addr 172.21.15.78:0/3121225635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81029e0 0x7fdfd8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 -- 172.21.15.78:0/3121225635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd81017f0 msgr2=0x7fdfd8197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd81017f0 0x7fdfd8197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 -- 172.21.15.78:0/3121225635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81029e0 msgr2=0x7fdfd8197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81029e0 0x7fdfd8197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd77fe640 1 -- 172.21.15.78:0/3121225635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd819f680 con 0x7fdfd81033a0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.898+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd81017f0 0x7fdfd8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.899+0000 7fdfd77fe640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81033a0 0x7fdfd819f010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000e970 tx=0x7fdfd000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.899+0000 7fdfbffff640 1 -- 172.21.15.78:0/3121225635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd000ccc0 con 0x7fdfd81033a0 2024-03-16T11:05:24.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.899+0000 7fdfbffff640 1 -- 172.21.15.78:0/3121225635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfd000ce60 con 0x7fdfd81033a0 2024-03-16T11:05:24.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.899+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd819f970 con 0x7fdfd81033a0 2024-03-16T11:05:24.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.899+0000 7fdfbffff640 1 -- 172.21.15.78:0/3121225635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0010430 con 0x7fdfd81033a0 2024-03-16T11:05:24.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.899+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd819fec0 con 0x7fdfd81033a0 2024-03-16T11:05:24.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfbffff640 1 -- 172.21.15.78:0/3121225635 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fdfd0004f90 con 0x7fdfd81033a0 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 Processor -- start 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 -- start start 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05afe0 0x55dd3c05b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0604d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c062fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd3bfbac80 con 0x55dd3c05afe0 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd3bf63aa0 con 0x55dd3c05b9d0 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfe0205740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd3bfb6d20 con 0x55dd3c060b40 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05afe0 0x55dd3c05b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfbffff640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2890 0x7fdfc00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05afe0 0x55dd3c05b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42982/0 (socket says 172.21.15.78:42982) 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.901+0000 7fdfd67fc640 1 -- 172.21.15.78:0/3536653742 learned_addr learned my addr 172.21.15.78:0/3536653742 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd77fe640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2890 0x7fdfc00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c062fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd67fc640 1 -- 172.21.15.78:0/3536653742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 msgr2=0x55dd3c062fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c062fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfbffff640 1 -- 172.21.15.78:0/3121225635 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdfd00cef90 con 0x7fdfd81033a0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd67fc640 1 -- 172.21.15.78:0/3536653742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05b9d0 msgr2=0x55dd3c0604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0604d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd67fc640 1 -- 172.21.15.78:0/3536653742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd3bf85ff0 con 0x55dd3c05afe0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05afe0 0x55dd3c05b400 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0135a0 tx=0x7fdfcc005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e12220ff8e7a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfd77fe640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2890 0x7fdfc00a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0097c30 tx=0x7fdfd006e500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfbd7fa640 1 -- 172.21.15.78:0/3536653742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc005020 con 0x55dd3c05afe0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfbd7fa640 1 -- 172.21.15.78:0/3536653742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfcc0047a0 con 0x55dd3c05afe0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.902+0000 7fdfbd7fa640 1 -- 172.21.15.78:0/3536653742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00f7e0 con 0x55dd3c05afe0 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- 172.21.15.78:0/3536653742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05afe0 msgr2=0x55dd3c05b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05afe0 0x55dd3c05b400 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0135a0 tx=0x7fdfcc005a20 comp rx=0 tx=0).stop 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- 172.21.15.78:0/3536653742 shutdown_connections 2024-03-16T11:05:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c062fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3536653742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05afe0 0x55dd3c05b400 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- 172.21.15.78:0/3536653742 >> 172.21.15.78:0/3536653742 conn(0x55dd3c052140 msgr2=0x55dd3c052540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- 172.21.15.78:0/3536653742 shutdown_connections 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- 172.21.15.78:0/3536653742 wait complete. 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- start start 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 Processor -- start 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c0650b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0655f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c063a90 con 0x55dd3c05b9d0 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd3bf76ef0 con 0x55dd3c05afe0 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c063c10 con 0x55dd3c060b40 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.903+0000 7fdfe0205740 10 client.0 _ll_register_callbacks cb 0x55dd3bf71dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c0650b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0655f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42984/0 (socket says 172.21.15.78:42984) 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2467278440 learned_addr learned my addr 172.21.15.78:0/2467278440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2467278440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 msgr2=0x55dd3c06c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2467278440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 msgr2=0x55dd3c0650b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c0650b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2467278440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd3bf83a30 con 0x55dd3c05b9d0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0655f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8004af0 tx=0x7fdfc8033800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8c44dc5dc96d35 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfbaff5640 1 -- 172.21.15.78:0/2467278440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8003970 con 0x55dd3c05b9d0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfbaff5640 1 -- 172.21.15.78:0/2467278440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdfc8003b10 con 0x55dd3c05b9d0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfcc0047a0 con 0x55dd3c05b9d0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfbaff5640 1 -- 172.21.15.78:0/2467278440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdfc8003d10 con 0x55dd3c05b9d0 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfe0205740 10 client.15111 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dd3bccf2d0 con 0x55dd3c05b9d0 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfbaff5640 1 -- 172.21.15.78:0/2467278440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfcc0047a0 con 0x55dd3c05b9d0 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.904+0000 7fdfbaff5640 1 -- 172.21.15.78:0/2467278440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fdfc80044d0 con 0x55dd3c05b9d0 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfe0205740 10 client.15111 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfe0205740 10 client.15111 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfe0205740 20 client.15111 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfe0205740 4 client.15111 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd3bfc81c0 con 0x55dd3c071b60 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfbaff5640 10 client.15111 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.905+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c072fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c5b2e0c5aed1a72 server_cookie=6cc893fb95b63107 in_seq=0 out_seq=0 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfbaff5640 1 -- 172.21.15.78:0/2467278440 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd3bfc81c0 con 0x55dd3c071b60 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfbaff5640 10 client.15111 handle_command_reply: tid=0 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 client.15111 shutdown 2024-03-16T11:05:24.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 msgr2=0x55dd3c0655f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0655f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8004af0 tx=0x7fdfc8033800 comp rx=0 tx=0).stop 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 shutdown_connections 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c072fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c0655f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfbaff5640 0 client.15111 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2467278440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c0650b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 >> 172.21.15.78:0/2467278440 conn(0x55dd3c052140 msgr2=0x55dd3c051380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 shutdown_connections 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 1 -- 172.21.15.78:0/2467278440 wait complete. 2024-03-16T11:05:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.906+0000 7fdfe0205740 20 client.15111 trim_cache size 0 max 16384 2024-03-16T11:05:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 Processor -- start 2024-03-16T11:05:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 -- start start 2024-03-16T11:05:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c05afe0 0x55dd3c065080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c06c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c060b40 0x55dd3c0655c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c071310 con 0x55dd3c05b9d0 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c0719e0 con 0x55dd3c060b40 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfe0205740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c0720b0 con 0x55dd3c05afe0 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c06c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c06c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42998/0 (socket says 172.21.15.78:42998) 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2244132733 learned_addr learned my addr 172.21.15.78:0/2244132733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c05afe0 0x55dd3c065080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c060b40 0x55dd3c0655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2244132733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c05afe0 msgr2=0x55dd3c065080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c05afe0 0x55dd3c065080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2244132733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c060b40 msgr2=0x55dd3c0655c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c060b40 0x55dd3c0655c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/2244132733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd3bd996b0 con 0x55dd3c05b9d0 2024-03-16T11:05:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c05afe0 0x55dd3c065080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c06c5f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdfc802c430 tx=0x7fdfc802bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444bc7e1053136b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.924+0000 7fdfbcff9640 1 -- 172.21.15.78:0/2244132733 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8034600 con 0x55dd3c05b9d0 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfbcff9640 1 -- 172.21.15.78:0/2244132733 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc8003970 con 0x55dd3c05b9d0 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfbcff9640 1 -- 172.21.15.78:0/2244132733 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc802c890 con 0x55dd3c05b9d0 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 -- 172.21.15.78:0/2244132733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 msgr2=0x55dd3c06c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c06c5f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdfc802c430 tx=0x7fdfc802bef0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 -- 172.21.15.78:0/2244132733 shutdown_connections 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c060b40 0x55dd3c0655c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c06c5f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 --2- 172.21.15.78:0/2244132733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c05afe0 0x55dd3c065080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 -- 172.21.15.78:0/2244132733 >> 172.21.15.78:0/2244132733 conn(0x55dd3c052140 msgr2=0x55dd3c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 -- 172.21.15.78:0/2244132733 shutdown_connections 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 -- 172.21.15.78:0/2244132733 wait complete. 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 -- start start 2024-03-16T11:05:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.925+0000 7fdfe0205740 1 Processor -- start 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c068120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c068660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c072e50 con 0x55dd3c05b9d0 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c072780 con 0x55dd3c05afe0 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd3c073520 con 0x55dd3c060b40 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 10 client.0 _ll_register_callbacks cb 0x7fdfc8009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c068660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c068660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43014/0 (socket says 172.21.15.78:43014) 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/3902537207 learned_addr learned my addr 172.21.15.78:0/3902537207 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c068120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/3902537207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 msgr2=0x55dd3c06b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 msgr2=0x55dd3c068120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c068120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 -- 172.21.15.78:0/3902537207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd3c06bc70 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfd6ffd640 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c068660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8034960 tx=0x7fdfc8034730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b1b35774255ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfbaff5640 1 -- 172.21.15.78:0/3902537207 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8061060 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfbaff5640 1 -- 172.21.15.78:0/3902537207 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdfc8004d50 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc8003970 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfbaff5640 1 -- 172.21.15.78:0/3902537207 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdfc802c890 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 10 client.15117 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.926+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dd3c0d7d10 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfbaff5640 1 -- 172.21.15.78:0/3902537207 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc8003970 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfbaff5640 1 -- 172.21.15.78:0/3902537207 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fdfc8066070 con 0x55dd3c05b9d0 2024-03-16T11:05:24.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfe0205740 10 client.15117 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfe0205740 10 client.15117 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfe0205740 20 client.15117 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c0db270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfe0205740 4 client.15117 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "status"}) -- 0x55dd3bfc2510 con 0x55dd3c071b60 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c0db270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfbaff5640 10 client.15117 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfd67fc640 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c0db270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fed1f169173c9c84 server_cookie=a5388854301d1af2 in_seq=0 out_seq=0 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfbaff5640 1 -- 172.21.15.78:0/3902537207 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55dd3bfc2510 con 0x55dd3c071b60 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.927+0000 7fdfbaff5640 10 client.15117 handle_command_reply: tid=0 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 client.15117 shutdown 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 msgr2=0x55dd3c068660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c068660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8034960 tx=0x7fdfc8034730 comp rx=0 tx=0).stop 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 shutdown_connections 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd3c071b60 0x55dd3c0db270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3c060b40 0x55dd3c06b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd3c05b9d0 0x55dd3c068660 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfbaff5640 0 client.15117 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 --2- 172.21.15.78:0/3902537207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd3c05afe0 0x55dd3c068120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 >> 172.21.15.78:0/3902537207 conn(0x55dd3c052140 msgr2=0x55dd3c065080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 shutdown_connections 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 1 -- 172.21.15.78:0/3902537207 wait complete. 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfe0205740 20 client.15117 trim_cache size 0 max 16384 2024-03-16T11:05:24.935 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24521, 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 84.645254592000001, 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 21, 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 90, 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 88, 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 89.972716004000006 2024-03-16T11:05:24.936 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:05:24.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2890 msgr2=0x7fdfc00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2890 0x7fdfc00a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0097c30 tx=0x7fdfd006e500 comp rx=0 tx=0).stop 2024-03-16T11:05:24.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81033a0 msgr2=0x7fdfd819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:24.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81033a0 0x7fdfd819f010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000e970 tx=0x7fdfd000ed70 comp rx=0 tx=0).stop 2024-03-16T11:05:24.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 shutdown_connections 2024-03-16T11:05:24.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81033a0 0x7fdfd819f010 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81029e0 0x7fdfd8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2890 0x7fdfc00a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 --2- 172.21.15.78:0/3121225635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd81017f0 0x7fdfd8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:24.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 >> 172.21.15.78:0/3121225635 conn(0x7fdfd80fcd80 msgr2=0x7fdfd8105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:24.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 shutdown_connections 2024-03-16T11:05:24.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:24.928+0000 7fdfdd61f640 1 -- 172.21.15.78:0/3121225635 wait complete. 2024-03-16T11:05:24.940 INFO:tasks.cephadm:Setup complete, yielding 2024-03-16T11:05:24.940 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T11:05:24.950 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T11:05:24.950 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-16T11:05:25.190 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:25.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:25 smithi078 ceph-mon[32235]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:25.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:25 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1396101285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:25 smithi098 ceph-mon[35848]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:25 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1396101285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:25 smithi153 ceph-mon[35521]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:25 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1396101285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:26.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 -- 172.21.15.78:0/3448138472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4103440 msgr2=0x7f06a41038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:26.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 --2- 172.21.15.78:0/3448138472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4103440 0x7f06a41038a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f06980099c0 tx=0x7f0698009cd0 comp rx=0 tx=0).stop 2024-03-16T11:05:26.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 -- 172.21.15.78:0/3448138472 shutdown_connections 2024-03-16T11:05:26.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 --2- 172.21.15.78:0/3448138472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a4102240 0x7f06a4102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:26.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 --2- 172.21.15.78:0/3448138472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a4103de0 0x7f06a410a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:26.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 --2- 172.21.15.78:0/3448138472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4103440 0x7f06a41038a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:26.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 -- 172.21.15.78:0/3448138472 >> 172.21.15.78:0/3448138472 conn(0x7f06a40fd7e0 msgr2=0x7f06a40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 -- 172.21.15.78:0/3448138472 shutdown_connections 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.834+0000 7f06a959d700 1 -- 172.21.15.78:0/3448138472 wait complete. 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 Processor -- start 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 -- start start 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4102240 0x7f06a419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a4103440 0x7f06a419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a4103de0 0x7f06a41a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f06a410dd00 con 0x7f06a4102240 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06a410db80 con 0x7f06a4103de0 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.835+0000 7f06a959d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06a410de80 con 0x7f06a4103440 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4102240 0x7f06a419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a4103440 0x7f06a419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4102240 0x7f06a419a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37010/0 (socket says 172.21.15.78:37010) 2024-03-16T11:05:26.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 -- 172.21.15.78:0/40725755 learned_addr learned my addr 172.21.15.78:0/40725755 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a8d9c700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a4103de0 0x7f06a41a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 -- 172.21.15.78:0/40725755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a4103440 msgr2=0x7f06a419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a4103440 0x7f06a419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 -- 172.21.15.78:0/40725755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a4103de0 msgr2=0x7f06a41a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a4103de0 0x7f06a41a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 -- 172.21.15.78:0/40725755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a41a21c0 con 0x7f06a4102240 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a37fe700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a4103440 0x7f06a419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a8d9c700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a4103de0 0x7f06a41a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a3fff700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4102240 0x7f06a419a560 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f069400e9a0 tx=0x7f069400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a17fa700 1 -- 172.21.15.78:0/40725755 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694004220 con 0x7f06a4102240 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a17fa700 1 -- 172.21.15.78:0/40725755 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06940043c0 con 0x7f06a4102240 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a41a24b0 con 0x7f06a4102240 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a17fa700 1 -- 172.21.15.78:0/40725755 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694003ce0 con 0x7f06a4102240 2024-03-16T11:05:26.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.836+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a41a2a40 con 0x7f06a4102240 2024-03-16T11:05:26.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.837+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a4068f90 con 0x7f06a4102240 2024-03-16T11:05:26.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.838+0000 7f06a17fa700 1 -- 172.21.15.78:0/40725755 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f069400c810 con 0x7f06a4102240 2024-03-16T11:05:26.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.839+0000 7f06a17fa700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f068c0a29a0 0x7f068c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:26.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.839+0000 7f06a37fe700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f068c0a29a0 0x7f068c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:26.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.839+0000 7f06a17fa700 1 -- 172.21.15.78:0/40725755 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f06940ced90 con 0x7f06a4102240 2024-03-16T11:05:26.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.839+0000 7f06a37fe700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f068c0a29a0 0x7f068c0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f0698005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:26.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:26.843+0000 7f06a17fa700 1 -- 172.21.15.78:0/40725755 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f069409bc90 con 0x7f06a4102240 2024-03-16T11:05:27.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.056+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f06a410a300 con 0x7f068c0a29a0 2024-03-16T11:05:27.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.057+0000 7f06a17fa700 1 -- 172.21.15.78:0/40725755 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f06a410a300 con 0x7f068c0a29a0 2024-03-16T11:05:27.058 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: cephadm 2024-03-16T11:05:27.058 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2024-03-16T11:05:27.058 INFO:teuthology.orchestra.run.smithi078.stdout:Paused: No 2024-03-16T11:05:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f068c0a29a0 msgr2=0x7f068c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f068c0a29a0 0x7f068c0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f0698005fb0 comp rx=0 tx=0).stop 2024-03-16T11:05:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4102240 msgr2=0x7f06a419a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4102240 0x7f06a419a560 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f069400e9a0 tx=0x7f069400ecb0 comp rx=0 tx=0).stop 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 shutdown_connections 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f068c0a29a0 0x7f068c0a4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a4103de0 0x7f06a41a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06a4102240 0x7f06a419a560 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 --2- 172.21.15.78:0/40725755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a4103440 0x7f06a419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 >> 172.21.15.78:0/40725755 conn(0x7f06a40fd7e0 msgr2=0x7f06a4108be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 shutdown_connections 2024-03-16T11:05:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:27.060+0000 7f06a959d700 1 -- 172.21.15.78:0/40725755 wait complete. 2024-03-16T11:05:27.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:27 smithi078 ceph-mon[32235]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:05:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:27 smithi098 ceph-mon[35848]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:05:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:27 smithi153 ceph-mon[35521]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:05:27.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-16T11:05:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:28.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:28 smithi078 ceph-mon[32235]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:28.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:28 smithi098 ceph-mon[35848]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:28.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:28 smithi153 ceph-mon[35521]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:29.307 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:29 smithi078 ceph-mon[32235]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:05:29.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:29 smithi098 ceph-mon[35848]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:05:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 -- 172.21.15.78:0/547419850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc103ea0 msgr2=0x7f35dc10a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 --2- 172.21.15.78:0/547419850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc103ea0 0x7f35dc10a750 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b0d0 tx=0x7f35d400b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 -- 172.21.15.78:0/547419850 shutdown_connections 2024-03-16T11:05:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 --2- 172.21.15.78:0/547419850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35dc102300 0x7f35dc102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 --2- 172.21.15.78:0/547419850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc103ea0 0x7f35dc10a750 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 --2- 172.21.15.78:0/547419850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35dc103500 0x7f35dc103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 -- 172.21.15.78:0/547419850 >> 172.21.15.78:0/547419850 conn(0x7f35dc0fd8e0 msgr2=0x7f35dc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 -- 172.21.15.78:0/547419850 shutdown_connections 2024-03-16T11:05:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.516+0000 7f35e11c6700 1 -- 172.21.15.78:0/547419850 wait complete. 2024-03-16T11:05:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.517+0000 7f35e11c6700 1 Processor -- start 2024-03-16T11:05:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.517+0000 7f35e11c6700 1 -- start start 2024-03-16T11:05:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc102300 0x7f35dc076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35dc103500 0x7f35dc0746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35dc103ea0 0x7f35dc074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc10dd90 con 0x7f35dc102300 2024-03-16T11:05:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc10dc10 con 0x7f35dc103ea0 2024-03-16T11:05:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc10df10 con 0x7f35dc103500 2024-03-16T11:05:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc102300 0x7f35dc076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc102300 0x7f35dc076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37042/0 (socket says 172.21.15.78:37042) 2024-03-16T11:05:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 -- 172.21.15.78:0/3916039985 learned_addr learned my addr 172.21.15.78:0/3916039985 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35da59c700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35dc103500 0x7f35dc0746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35db59e700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35dc103ea0 0x7f35dc074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 -- 172.21.15.78:0/3916039985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35dc103500 msgr2=0x7f35dc0746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35dc103500 0x7f35dc0746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 -- 172.21.15.78:0/3916039985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35dc103ea0 msgr2=0x7f35dc074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35dc103ea0 0x7f35dc074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 -- 172.21.15.78:0/3916039985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35dc0754b0 con 0x7f35dc102300 2024-03-16T11:05:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35dad9d700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc102300 0x7f35dc076070 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f35d000e950 tx=0x7f35d000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35c3fff700 1 -- 172.21.15.78:0/3916039985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0004230 con 0x7f35dc102300 2024-03-16T11:05:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35dc1a2560 con 0x7f35dc102300 2024-03-16T11:05:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35c3fff700 1 -- 172.21.15.78:0/3916039985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d00043d0 con 0x7f35dc102300 2024-03-16T11:05:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35c3fff700 1 -- 172.21.15.78:0/3916039985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0003d60 con 0x7f35dc102300 2024-03-16T11:05:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.518+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35dc1a2a70 con 0x7f35dc102300 2024-03-16T11:05:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.519+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35dc068f90 con 0x7f35dc102300 2024-03-16T11:05:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.520+0000 7f35c3fff700 1 -- 172.21.15.78:0/3916039985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f35d000c7e0 con 0x7f35dc102300 2024-03-16T11:05:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.521+0000 7f35c3fff700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35c40a2920 0x7f35c40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.521+0000 7f35da59c700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35c40a2920 0x7f35c40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.521+0000 7f35c3fff700 1 -- 172.21.15.78:0/3916039985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f35d00cee70 con 0x7f35dc102300 2024-03-16T11:05:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.521+0000 7f35da59c700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35c40a2920 0x7f35c40a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009680 tx=0x7f35cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:29.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.526+0000 7f35c3fff700 1 -- 172.21.15.78:0/3916039985 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f35d009bdb0 con 0x7f35dc102300 2024-03-16T11:05:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:29 smithi153 ceph-mon[35521]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:05:29.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.736+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f35dc062710 con 0x7f35c40a2920 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.749+0000 7f35c3fff700 1 -- 172.21.15.78:0/3916039985 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f35dc062710 con 0x7f35c40a2920 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.a smithi078 running (97s) 18s ago 97s 13.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 67e9c56b4cf4 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.b smithi098 running (95s) 28s ago 94s 26.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 90329a530743 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.c smithi153 running (108s) 29s ago 108s 18.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 43af1a72d8d1 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.d smithi078 running (117s) 18s ago 116s 13.6M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 721c557738ef 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.e smithi098 running (114s) 28s ago 113s 19.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 84f36ef85f50 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.f smithi153 running (111s) 29s ago 111s 19.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 32a197345a73 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.g smithi078 running (100s) 18s ago 100s 13.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 6a18916af149 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.h smithi098 running (106s) 28s ago 105s 17.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 3fcbdba0c316 2024-03-16T11:05:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:mds.i smithi153 running (103s) 29s ago 103s 20.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 2c6276463a81 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mds.j smithi078 running (119s) 18s ago 119s 13.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b e69bad745878 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mds.k smithi098 running (2m) 28s ago 2m 16.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b c3d1d4e29730 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mds.l smithi153 running (2m) 29s ago 2m 14.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 9455dd4412f1 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi078 *:9283,8765 running (9m) 18s ago 9m 514M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 564b1a81a389 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi098 *:8443,8765 running (7m) 28s ago 7m 454M - 19.0.0-2162-g3d78f5ba be86dc8cb22b eef3db8b636f 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.z smithi153 *:8443,8765 running (7m) 29s ago 7m 451M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 50074f2f014f 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (9m) 18s ago 9m 53.2M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b b5c01b21f69c 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi098 running (7m) 28s ago 7m 41.9M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b bfbc5ea10822 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi153 running (7m) 29s ago 7m 38.4M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b c9ce64dc1ae8 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6m) 18s ago 6m 57.9M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b cfe5418bd94c 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi098 running (6m) 28s ago 6m 59.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b fe6388661d71 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi153 running (6m) 29s ago 6m 49.1M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 5253383d8cbc 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5m) 18s ago 5m 58.3M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8bc55634216e 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi098 running (5m) 28s ago 5m 44.6M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b f6e9968dc060 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi153 running (4m) 29s ago 4m 56.9M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b b630a4727adb 2024-03-16T11:05:29.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi078 running (4m) 18s ago 4m 44.7M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8c62e68d5738 2024-03-16T11:05:29.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi098 running (4m) 28s ago 4m 46.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 35bc74da04eb 2024-03-16T11:05:29.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi153 running (3m) 29s ago 3m 46.2M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 9d05dc8d1b0e 2024-03-16T11:05:29.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (3m) 18s ago 3m 55.4M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 4759b92067e5 2024-03-16T11:05:29.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi098 running (2m) 28s ago 2m 58.2M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 513be66d1c08 2024-03-16T11:05:29.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi153 running (2m) 29s ago 2m 43.7M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 557a5085e26b 2024-03-16T11:05:29.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35c40a2920 msgr2=0x7f35c40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:29.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35c40a2920 0x7f35c40a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009680 tx=0x7f35cc009260 comp rx=0 tx=0).stop 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc102300 msgr2=0x7f35dc076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc102300 0x7f35dc076070 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f35d000e950 tx=0x7f35d000ec60 comp rx=0 tx=0).stop 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 shutdown_connections 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35c40a2920 0x7f35c40a4de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35dc103ea0 0x7f35dc074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35dc102300 0x7f35dc076070 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 --2- 172.21.15.78:0/3916039985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35dc103500 0x7f35dc0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.751+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 >> 172.21.15.78:0/3916039985 conn(0x7f35dc0fd8e0 msgr2=0x7f35dc106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.752+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 shutdown_connections 2024-03-16T11:05:29.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:29.752+0000 7f35e11c6700 1 -- 172.21.15.78:0/3916039985 wait complete. 2024-03-16T11:05:30.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-16T11:05:30.678 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:31 smithi098 ceph-mon[35848]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:31 smithi098 ceph-mon[35848]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:31 smithi153 ceph-mon[35521]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:31 smithi153 ceph-mon[35521]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:31 smithi078 ceph-mon[32235]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:31 smithi078 ceph-mon[32235]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:32.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 -- 172.21.15.78:0/3033532485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103500 msgr2=0x7ff964103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 --2- 172.21.15.78:0/3033532485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103500 0x7ff964103960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff95800b0d0 tx=0x7ff95800b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 -- 172.21.15.78:0/3033532485 shutdown_connections 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 --2- 172.21.15.78:0/3033532485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964102300 0x7ff964102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 --2- 172.21.15.78:0/3033532485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964103ea0 0x7ff96410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 --2- 172.21.15.78:0/3033532485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103500 0x7ff964103960 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 -- 172.21.15.78:0/3033532485 >> 172.21.15.78:0/3033532485 conn(0x7ff9640fd8e0 msgr2=0x7ff9640ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 -- 172.21.15.78:0/3033532485 shutdown_connections 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.248+0000 7ff9693b7700 1 -- 172.21.15.78:0/3033532485 wait complete. 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 Processor -- start 2024-03-16T11:05:32.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 -- start start 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964102300 0x7ff9641984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964103500 0x7ff964198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103ea0 0x7ff96419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff96410ddf0 con 0x7ff964103ea0 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff96410dc70 con 0x7ff964102300 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9693b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff96410df70 con 0x7ff964103500 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9637fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103ea0 0x7ff96419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9637fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103ea0 0x7ff96419faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37052/0 (socket says 172.21.15.78:37052) 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff962ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964102300 0x7ff9641984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9637fe700 1 -- 172.21.15.78:0/75636434 learned_addr learned my addr 172.21.15.78:0/75636434 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.249+0000 7ff9627fc700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964103500 0x7ff964198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff962ffd700 1 -- 172.21.15.78:0/75636434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964103500 msgr2=0x7ff964198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff962ffd700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964103500 0x7ff964198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff962ffd700 1 -- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103ea0 msgr2=0x7ff96419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff962ffd700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103ea0 0x7ff96419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff962ffd700 1 -- 172.21.15.78:0/75636434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9641a0140 con 0x7ff964102300 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff9637fe700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103ea0 0x7ff96419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff962ffd700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964102300 0x7ff9641984e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00e9c0 tx=0x7ff95c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff94bfff700 1 -- 172.21.15.78:0/75636434 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c004230 con 0x7ff964102300 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff94bfff700 1 -- 172.21.15.78:0/75636434 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff95c0043d0 con 0x7ff964102300 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9641a0430 con 0x7ff964102300 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff94bfff700 1 -- 172.21.15.78:0/75636434 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c003d90 con 0x7ff964102300 2024-03-16T11:05:32.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.250+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9641a0920 con 0x7ff964102300 2024-03-16T11:05:32.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.251+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff964068f90 con 0x7ff964102300 2024-03-16T11:05:32.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.252+0000 7ff94bfff700 1 -- 172.21.15.78:0/75636434 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff95c00c7c0 con 0x7ff964102300 2024-03-16T11:05:32.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.252+0000 7ff94bfff700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff94c0a2920 0x7ff94c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:32.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.253+0000 7ff9627fc700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff94c0a2920 0x7ff94c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:32.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.253+0000 7ff94bfff700 1 -- 172.21.15.78:0/75636434 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff95c0ceeb0 con 0x7ff964102300 2024-03-16T11:05:32.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.253+0000 7ff9627fc700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff94c0a2920 0x7ff94c0a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff958009e60 tx=0x7ff95800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:32.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.257+0000 7ff94bfff700 1 -- 172.21.15.78:0/75636434 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff95c09bdf0 con 0x7ff964102300 2024-03-16T11:05:32.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.461+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ff964062710 con 0x7ff94c0a2920 2024-03-16T11:05:32.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.467+0000 7ff94bfff700 1 -- 172.21.15.78:0/75636434 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ff964062710 con 0x7ff94c0a2920 2024-03-16T11:05:32.468 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-16T11:05:32.468 INFO:teuthology.orchestra.run.smithi078.stdout:mds.all 12/12 31s ago 2m smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-03-16T11:05:32.468 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 3/3 31s ago 7m smithi078=x;smithi098=y;smithi153=z;count:3 2024-03-16T11:05:32.468 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 31s ago 7m smithi078:172.21.15.78=a;smithi098:172.21.15.98=b;smithi153:172.21.15.153=c;count:3 2024-03-16T11:05:32.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd 12 31s ago - 2024-03-16T11:05:32.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff94c0a2920 msgr2=0x7ff94c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:32.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff94c0a2920 0x7ff94c0a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff958009e60 tx=0x7ff95800b040 comp rx=0 tx=0).stop 2024-03-16T11:05:32.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964102300 msgr2=0x7ff9641984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:32.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964102300 0x7ff9641984e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00e9c0 tx=0x7ff95c00ecd0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 shutdown_connections 2024-03-16T11:05:32.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff94c0a2920 0x7ff94c0a4de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff964103ea0 0x7ff96419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964102300 0x7ff9641984e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 --2- 172.21.15.78:0/75636434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964103500 0x7ff964198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:32.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 >> 172.21.15.78:0/75636434 conn(0x7ff9640fd8e0 msgr2=0x7ff964106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:32.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 shutdown_connections 2024-03-16T11:05:32.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:32.470+0000 7ff9693b7700 1 -- 172.21.15.78:0/75636434 wait complete. 2024-03-16T11:05:33.071 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-16T11:05:33.313 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:33.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:33 smithi078 ceph-mon[32235]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:33.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:33 smithi098 ceph-mon[35848]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:33 smithi153 ceph-mon[35521]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:34.375 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:34 smithi078 ceph-mon[32235]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:34.376 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:34 smithi078 ceph-mon[32235]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-16T11:05:34.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:34 smithi098 ceph-mon[35848]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:34.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:34 smithi098 ceph-mon[35848]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-16T11:05:34.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:34 smithi153 ceph-mon[35521]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:34.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:34 smithi153 ceph-mon[35521]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-16T11:05:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 -- 172.21.15.78:0/3418832727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 msgr2=0x7fe784103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 --2- 172.21.15.78:0/3418832727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784103880 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe7700099c0 tx=0x7fe770009cd0 comp rx=0 tx=0).stop 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 -- 172.21.15.78:0/3418832727 shutdown_connections 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 --2- 172.21.15.78:0/3418832727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe784102220 0x7fe784102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 --2- 172.21.15.78:0/3418832727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe784103dc0 0x7fe78410a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 --2- 172.21.15.78:0/3418832727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784103880 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 -- 172.21.15.78:0/3418832727 >> 172.21.15.78:0/3418832727 conn(0x7fe7840fd7e0 msgr2=0x7fe7840ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 -- 172.21.15.78:0/3418832727 shutdown_connections 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.839+0000 7fe78999e700 1 -- 172.21.15.78:0/3418832727 wait complete. 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 Processor -- start 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 -- start start 2024-03-16T11:05:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe784102220 0x7fe784198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe784103dc0 0x7fe78419f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe78410dc90 con 0x7fe784103420 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe78410db10 con 0x7fe784103dc0 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78999e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe78410de10 con 0x7fe784102220 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37076/0 (socket says 172.21.15.78:37076) 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 -- 172.21.15.78:0/270741689 learned_addr learned my addr 172.21.15.78:0/270741689 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78899c700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe784102220 0x7fe784198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe78919d700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe784103dc0 0x7fe78419f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 -- 172.21.15.78:0/270741689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe784102220 msgr2=0x7fe784198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe784102220 0x7fe784198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 -- 172.21.15.78:0/270741689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe784103dc0 msgr2=0x7fe78419f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe784103dc0 0x7fe78419f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 -- 172.21.15.78:0/270741689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78419ff60 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe783fff700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784198840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe770000c00 tx=0x7fe7700057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.840+0000 7fe781ffb700 1 -- 172.21.15.78:0/270741689 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770027070 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.841+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7841a01f0 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.841+0000 7fe781ffb700 1 -- 172.21.15.78:0/270741689 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe770004050 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.841+0000 7fe781ffb700 1 -- 172.21.15.78:0/270741689 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770022ce0 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.841+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7841a0640 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.842+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe784068f90 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.843+0000 7fe781ffb700 1 -- 172.21.15.78:0/270741689 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe77002a070 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.843+0000 7fe781ffb700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe7740a2890 0x7fe7740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.843+0000 7fe78899c700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe7740a2890 0x7fe7740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.843+0000 7fe781ffb700 1 -- 172.21.15.78:0/270741689 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe7700cf950 con 0x7fe784103420 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.844+0000 7fe78899c700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe7740a2890 0x7fe7740a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe784103280 tx=0x7fe778007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:34.848+0000 7fe781ffb700 1 -- 172.21.15.78:0/270741689 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe77005e740 con 0x7fe784103420 2024-03-16T11:05:35.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.051+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fe784062710 con 0x7fe7740a2890 2024-03-16T11:05:35.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.052+0000 7fe781ffb700 1 -- 172.21.15.78:0/270741689 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fe784062710 con 0x7fe7740a2890 2024-03-16T11:05:35.053 INFO:teuthology.orchestra.run.smithi078.stdout:HOST ADDR LABELS STATUS 2024-03-16T11:05:35.053 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 172.21.15.78 2024-03-16T11:05:35.053 INFO:teuthology.orchestra.run.smithi078.stdout:smithi098 172.21.15.98 2024-03-16T11:05:35.053 INFO:teuthology.orchestra.run.smithi078.stdout:smithi153 172.21.15.153 2024-03-16T11:05:35.053 INFO:teuthology.orchestra.run.smithi078.stdout:3 hosts in cluster 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe7740a2890 msgr2=0x7fe7740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe7740a2890 0x7fe7740a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe784103280 tx=0x7fe778007400 comp rx=0 tx=0).stop 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 msgr2=0x7fe784198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784198840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe770000c00 tx=0x7fe7700057c0 comp rx=0 tx=0).stop 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 shutdown_connections 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe7740a2890 0x7fe7740a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe784102220 0x7fe784198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe784103dc0 0x7fe78419f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 --2- 172.21.15.78:0/270741689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe784103420 0x7fe784198840 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 >> 172.21.15.78:0/270741689 conn(0x7fe7840fd7e0 msgr2=0x7fe7841067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:35.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 shutdown_connections 2024-03-16T11:05:35.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:35.054+0000 7fe78999e700 1 -- 172.21.15.78:0/270741689 wait complete. 2024-03-16T11:05:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:35 smithi078 ceph-mon[32235]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:05:35.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:35 smithi078 ceph-mon[32235]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:35 smithi098 ceph-mon[35848]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:05:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:35 smithi098 ceph-mon[35848]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:35 smithi153 ceph-mon[35521]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:05:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:35 smithi153 ceph-mon[35521]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:35.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-16T11:05:35.993 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:36.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:36 smithi078 ceph-mon[32235]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:05:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:36 smithi098 ceph-mon[35848]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:05:36.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:36 smithi153 ceph-mon[35521]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:05:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 -- 172.21.15.78:0/2468207308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70281022d0 msgr2=0x7f70281026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 --2- 172.21.15.78:0/2468207308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70281022d0 0x7f70281026f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f701c0099c0 tx=0x7f701c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 -- 172.21.15.78:0/2468207308 shutdown_connections 2024-03-16T11:05:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 --2- 172.21.15.78:0/2468207308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7028103e70 0x7f702810a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 --2- 172.21.15.78:0/2468207308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70281022d0 0x7f70281026f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 --2- 172.21.15.78:0/2468207308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70281034d0 0x7f7028103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 -- 172.21.15.78:0/2468207308 >> 172.21.15.78:0/2468207308 conn(0x7f70280fd8b0 msgr2=0x7f70280ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:37.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 -- 172.21.15.78:0/2468207308 shutdown_connections 2024-03-16T11:05:37.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.685+0000 7f702caa9700 1 -- 172.21.15.78:0/2468207308 wait complete. 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 Processor -- start 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 -- start start 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70281022d0 0x7f7028198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70281034d0 0x7f7028198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7028103e70 0x7f702819f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f702810dd40 con 0x7f7028103e70 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f702810dbc0 con 0x7f70281034d0 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702caa9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f702810dec0 con 0x7f70281022d0 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f7026d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7028103e70 0x7f702819f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f7026d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7028103e70 0x7f702819f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49996/0 (socket says 172.21.15.78:49996) 2024-03-16T11:05:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f7026d9d700 1 -- 172.21.15.78:0/3590256972 learned_addr learned my addr 172.21.15.78:0/3590256972 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f702659c700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70281022d0 0x7f7028198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.686+0000 7f7025d9b700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70281034d0 0x7f7028198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f7026d9d700 1 -- 172.21.15.78:0/3590256972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70281022d0 msgr2=0x7f7028198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f7026d9d700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70281022d0 0x7f7028198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f7026d9d700 1 -- 172.21.15.78:0/3590256972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70281034d0 msgr2=0x7f7028198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f7026d9d700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70281034d0 0x7f7028198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f7026d9d700 1 -- 172.21.15.78:0/3590256972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70281a0090 con 0x7f7028103e70 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f7026d9d700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7028103e70 0x7f702819f9f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7020007f10 tx=0x7f702000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f70177fe700 1 -- 172.21.15.78:0/3590256972 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020004bb0 con 0x7f7028103e70 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7028074560 con 0x7f7028103e70 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f70177fe700 1 -- 172.21.15.78:0/3590256972 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7020004d50 con 0x7f7028103e70 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f70177fe700 1 -- 172.21.15.78:0/3590256972 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70200058e0 con 0x7f7028103e70 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.687+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7028074a50 con 0x7f7028103e70 2024-03-16T11:05:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.688+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7028068f90 con 0x7f7028103e70 2024-03-16T11:05:37.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.689+0000 7f70177fe700 1 -- 172.21.15.78:0/3590256972 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f702001a020 con 0x7f7028103e70 2024-03-16T11:05:37.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.690+0000 7f70177fe700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f70100a2890 0x7f70100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:37.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.690+0000 7f702659c700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f70100a2890 0x7f70100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:37.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.690+0000 7f70177fe700 1 -- 172.21.15.78:0/3590256972 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f70200ce160 con 0x7f7028103e70 2024-03-16T11:05:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.690+0000 7f702659c700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f70100a2890 0x7f70100a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f701c00b580 tx=0x7f701c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.694+0000 7f70177fe700 1 -- 172.21.15.78:0/3590256972 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7020010070 con 0x7f7028103e70 2024-03-16T11:05:37.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.902+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f7028062710 con 0x7f70100a2890 2024-03-16T11:05:37.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.904+0000 7f70177fe700 1 -- 172.21.15.78:0/3590256972 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f7028062710 con 0x7f70100a2890 2024-03-16T11:05:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-16T11:05:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QR400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T11:05:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-16T11:05:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-16T11:05:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f70100a2890 msgr2=0x7f70100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f70100a2890 0x7f70100a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f701c00b580 tx=0x7f701c005fb0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7028103e70 msgr2=0x7f702819f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7028103e70 0x7f702819f9f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7020007f10 tx=0x7f702000d3b0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 shutdown_connections 2024-03-16T11:05:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f70100a2890 0x7f70100a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70281022d0 0x7f7028198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7028103e70 0x7f702819f9f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 --2- 172.21.15.78:0/3590256972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70281034d0 0x7f7028198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:37.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.906+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 >> 172.21.15.78:0/3590256972 conn(0x7f70280fd8b0 msgr2=0x7f70281068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:37.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.907+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 shutdown_connections 2024-03-16T11:05:37.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:37.907+0000 7f702caa9700 1 -- 172.21.15.78:0/3590256972 wait complete. 2024-03-16T11:05:38.745 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T11:05:38.754 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T11:05:38.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-16T11:05:39.004 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:39 smithi153 ceph-mon[35521]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:39 smithi153 ceph-mon[35521]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-16T11:05:39.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:39 smithi078 ceph-mon[32235]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:39.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:39 smithi078 ceph-mon[32235]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-16T11:05:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:39 smithi098 ceph-mon[35848]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:39 smithi098 ceph-mon[35848]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 -- 172.21.15.78:0/2292452575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0102dd0 msgr2=0x7f4df0109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/2292452575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0102dd0 0x7f4df0109680 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4de800b0d0 tx=0x7f4de800b3e0 comp rx=0 tx=0).stop 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 -- 172.21.15.78:0/2292452575 shutdown_connections 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/2292452575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0101230 0x7f4df0101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/2292452575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0102dd0 0x7f4df0109680 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/2292452575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df0102430 0x7f4df0102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 -- 172.21.15.78:0/2292452575 >> 172.21.15.78:0/2292452575 conn(0x7f4df00fc7f0 msgr2=0x7f4df00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 -- 172.21.15.78:0/2292452575 shutdown_connections 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.817+0000 7f4df5d2b700 1 -- 172.21.15.78:0/2292452575 wait complete. 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 Processor -- start 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 -- start start 2024-03-16T11:05:40.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0101230 0x7f4df01051c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df0102430 0x7f4df0103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0102dd0 0x7f4df0103d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010cca0 con 0x7f4df0101230 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010cb20 con 0x7f4df0102430 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4df5d2b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010ce20 con 0x7f4df0102dd0 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4def7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0101230 0x7f4df01051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4def7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0101230 0x7f4df01051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50014/0 (socket says 172.21.15.78:50014) 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4def7fe700 1 -- 172.21.15.78:0/559357903 learned_addr learned my addr 172.21.15.78:0/559357903 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4de6ffd700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df0102430 0x7f4df0103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.818+0000 7f4deffff700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0102dd0 0x7f4df0103d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4def7fe700 1 -- 172.21.15.78:0/559357903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0102dd0 msgr2=0x7f4df0103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4def7fe700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0102dd0 0x7f4df0103d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4def7fe700 1 -- 172.21.15.78:0/559357903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df0102430 msgr2=0x7f4df0103810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:40.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4def7fe700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df0102430 0x7f4df0103810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4def7fe700 1 -- 172.21.15.78:0/559357903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df0104600 con 0x7f4df0101230 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4de6ffd700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df0102430 0x7f4df0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4def7fe700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0101230 0x7f4df01051c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4de000d980 tx=0x7f4de000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4ded7fa700 1 -- 172.21.15.78:0/559357903 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0016070 con 0x7f4df0101230 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df01a2500 con 0x7f4df0101230 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4ded7fa700 1 -- 172.21.15.78:0/559357903 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4de000fd40 con 0x7f4df0101230 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4ded7fa700 1 -- 172.21.15.78:0/559357903 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0019360 con 0x7f4df0101230 2024-03-16T11:05:40.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.819+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df01a29f0 con 0x7f4df0101230 2024-03-16T11:05:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.820+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df0068f90 con 0x7f4df0101230 2024-03-16T11:05:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.821+0000 7f4ded7fa700 1 -- 172.21.15.78:0/559357903 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4de0019070 con 0x7f4df0101230 2024-03-16T11:05:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.822+0000 7f4ded7fa700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4ddc0a2890 0x7f4ddc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.822+0000 7f4de6ffd700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4ddc0a2890 0x7f4ddc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.822+0000 7f4ded7fa700 1 -- 172.21.15.78:0/559357903 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f4de00ce1e0 con 0x7f4df0101230 2024-03-16T11:05:40.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.822+0000 7f4de6ffd700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4ddc0a2890 0x7f4ddc0a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009fd0 tx=0x7f4dd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:40.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:40.826+0000 7f4ded7fa700 1 -- 172.21.15.78:0/559357903 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4de009b280 con 0x7f4df0101230 2024-03-16T11:05:41.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.068+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f4df01a3440 con 0x7f4df0101230 2024-03-16T11:05:41.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.068+0000 7f4ded7fa700 1 -- 172.21.15.78:0/559357903 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3048 (secure 0 0 0) 0x7f4de009ab80 con 0x7f4df0101230 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:e21 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:legacy client fscid: 1 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:Filesystem 'cephfs' (1) 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:fs_name cephfs 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 21 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-03-16T11:03:59.617240+0000 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-03-16T11:04:03.658334+0000 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:tableserver 0 2024-03-16T11:05:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:root 0 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:session_timeout 300 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:session_autoclose 300 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:max_file_size 1099511627776 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:max_xattr_size 65536 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:required_client_features {} 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure 0 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure_osd_epoch 0 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:max_mds 3 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:in 0,1,2 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:up {0=24521,1=24505,2=24488} 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:failed 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:damaged 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:stopped 2024-03-16T11:05:41.072 INFO:teuthology.orchestra.run.smithi078.stdout:data_pools [3] 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:metadata_pool 2 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:inline_data disabled 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:balancer 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:bal_rank_mask -1 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:standby_count_wanted 0 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:qdb_cluster leader: 24521 members: 24488,24505,24521 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b{0:24521} state up:active seq 3 addr [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c{0:24484} state up:standby-replay seq 1 addr [v2:172.21.15.153:6836/3505861862,v1:172.21.15.153:6837/3505861862] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.i{1:24505} state up:active seq 5 addr [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.f{1:24472} state up:standby-replay seq 1 addr [v2:172.21.15.153:6834/1109965992,v1:172.21.15.153:6835/1109965992] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.h{2:24488} state up:active seq 6 addr [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.e{2:24458} state up:standby-replay seq 1 addr [v2:172.21.15.98:6834/1190197046,v1:172.21.15.98:6835/1190197046] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout:Standby daemons: 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.78:6834/1233514200,v1:172.21.15.78:6835/1233514200] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d{-1:14574} state up:standby seq 1 addr [v2:172.21.15.78:6836/4216287385,v1:172.21.15.78:6837/4216287385] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.g{-1:14643} state up:standby seq 1 addr [v2:172.21.15.78:6838/3781553715,v1:172.21.15.78:6839/3781553715] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a{-1:14655} state up:standby seq 1 addr [v2:172.21.15.78:6840/627084813,v1:172.21.15.78:6841/627084813] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.k{-1:24425} state up:standby seq 1 addr [v2:172.21.15.98:6832/3200006820,v1:172.21.15.98:6833/3200006820] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.074 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.l{-1:24436} state up:standby seq 1 addr [v2:172.21.15.153:6832/3909734389,v1:172.21.15.153:6833/3909734389] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.070+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4ddc0a2890 msgr2=0x7f4ddc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.070+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4ddc0a2890 0x7f4ddc0a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009fd0 tx=0x7f4dd8009260 comp rx=0 tx=0).stop 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.070+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0101230 msgr2=0x7f4df01051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.070+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0101230 0x7f4df01051c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4de000d980 tx=0x7f4de000dc90 comp rx=0 tx=0).stop 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 shutdown_connections 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4ddc0a2890 0x7f4ddc0a4d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df0102dd0 0x7f4df0103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df0101230 0x7f4df01051c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 --2- 172.21.15.78:0/559357903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df0102430 0x7f4df0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:41.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 >> 172.21.15.78:0/559357903 conn(0x7f4df00fc7f0 msgr2=0x7f4df00fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:41.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 shutdown_connections 2024-03-16T11:05:41.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:41.071+0000 7f4df5d2b700 1 -- 172.21.15.78:0/559357903 wait complete. 2024-03-16T11:05:41.076 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 21 2024-03-16T11:05:41.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:41 smithi078 ceph-mon[32235]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:05:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:41 smithi098 ceph-mon[35848]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:05:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:41 smithi153 ceph-mon[35521]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:05:41.844 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-16T11:05:42.090 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:42.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:42 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/559357903' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T11:05:42.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:42 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/559357903' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T11:05:42.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:42 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/559357903' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T11:05:43.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:43 smithi078 ceph-mon[32235]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:05:43.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:43 smithi098 ceph-mon[35848]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:05:43.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:43 smithi153 ceph-mon[35521]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:05:43.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 -- 172.21.15.78:0/369234288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103500 msgr2=0x7f5920103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 --2- 172.21.15.78:0/369234288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103500 0x7f5920103960 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f59140099c0 tx=0x7f5914009cd0 comp rx=0 tx=0).stop 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 -- 172.21.15.78:0/369234288 shutdown_connections 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 --2- 172.21.15.78:0/369234288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5920102300 0x7f5920102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 --2- 172.21.15.78:0/369234288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5920103ea0 0x7f592010a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 --2- 172.21.15.78:0/369234288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103500 0x7f5920103960 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 -- 172.21.15.78:0/369234288 >> 172.21.15.78:0/369234288 conn(0x7f59200fd8e0 msgr2=0x7f59200ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 -- 172.21.15.78:0/369234288 shutdown_connections 2024-03-16T11:05:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 -- 172.21.15.78:0/369234288 wait complete. 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 Processor -- start 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.797+0000 7f5924ab3700 1 -- start start 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f5924ab3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5920102300 0x7f592019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f5924ab3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5920103500 0x7f592019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f5924ab3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103ea0 0x7f59201a6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f5924ab3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5920075940 con 0x7f5920103ea0 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f5924ab3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59200757c0 con 0x7f5920103500 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f5924ab3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5920075ac0 con 0x7f5920102300 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103ea0 0x7f59201a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5920103500 0x7f592019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103ea0 0x7f59201a6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50022/0 (socket says 172.21.15.78:50022) 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5920102300 0x7f592019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591ed9d700 1 -- 172.21.15.78:0/2849333935 learned_addr learned my addr 172.21.15.78:0/2849333935 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591dd9b700 1 -- 172.21.15.78:0/2849333935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5920102300 msgr2=0x7f592019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591dd9b700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5920102300 0x7f592019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591dd9b700 1 -- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103ea0 msgr2=0x7f59201a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591dd9b700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103ea0 0x7f59201a6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591dd9b700 1 -- 172.21.15.78:0/2849333935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59201a66c0 con 0x7f5920103500 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591ed9d700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103ea0 0x7f59201a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f591dd9b700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5920103500 0x7f592019efa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f59140036f0 tx=0x7f591400f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.798+0000 7f590f7fe700 1 -- 172.21.15.78:0/2849333935 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914009e60 con 0x7f5920103500 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.799+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59201a6950 con 0x7f5920103500 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.799+0000 7f590f7fe700 1 -- 172.21.15.78:0/2849333935 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5914005940 con 0x7f5920103500 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.799+0000 7f590f7fe700 1 -- 172.21.15.78:0/2849333935 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914005450 con 0x7f5920103500 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.799+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59201a6e30 con 0x7f5920103500 2024-03-16T11:05:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.799+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5920068f90 con 0x7f5920103500 2024-03-16T11:05:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.801+0000 7f590f7fe700 1 -- 172.21.15.78:0/2849333935 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5914003b30 con 0x7f5920103500 2024-03-16T11:05:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.801+0000 7f590f7fe700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59080a2890 0x7f59080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.801+0000 7f591e59c700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59080a2890 0x7f59080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.801+0000 7f590f7fe700 1 -- 172.21.15.78:0/2849333935 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f59140cf6d0 con 0x7f5920103500 2024-03-16T11:05:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.801+0000 7f591e59c700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59080a2890 0x7f59080a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5920103360 tx=0x7f591800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:43.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:43.805+0000 7f590f7fe700 1 -- 172.21.15.78:0/2849333935 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5914013070 con 0x7f5920103500 2024-03-16T11:05:44.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.002+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f59201a7270 con 0x7f5920103500 2024-03-16T11:05:44.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.003+0000 7f590f7fe700 1 -- 172.21.15.78:0/2849333935 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5280 (secure 0 0 0) 0x7f59140213c0 con 0x7f5920103500 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 90 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:fsid a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-03-16T10:56:07.773806+0000 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-03-16T11:05:03.097866+0000 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 26 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2024-03-16T11:05:44.004 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client luminous 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release squid 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 12 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/467332989,v1:172.21.15.78:6803/467332989] [v2:172.21.15.78:6804/467332989,v1:172.21.15.78:6805/467332989] exists,up 543437a6-61d5-4969-be4e-7a20f1edc236 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/64709971,v1:172.21.15.98:6801/64709971] [v2:172.21.15.98:6802/64709971,v1:172.21.15.98:6803/64709971] exists,up 2c57f283-922d-4ef4-a00d-ce2f62a7e19a 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/2715466096,v1:172.21.15.153:6801/2715466096] [v2:172.21.15.153:6802/2715466096,v1:172.21.15.153:6803/2715466096] exists,up 4559999e-89a9-433d-b3b8-bdec51aebeb8 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 26 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6810/3489770309,v1:172.21.15.78:6811/3489770309] [v2:172.21.15.78:6812/3489770309,v1:172.21.15.78:6813/3489770309] exists,up e3a68ec8-b4f9-42fb-9f85-e683ef538fbe 2024-03-16T11:05:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 34 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/2809778583,v1:172.21.15.98:6809/2809778583] [v2:172.21.15.98:6810/2809778583,v1:172.21.15.98:6811/2809778583] exists,up 9d39555d-45cf-4507-80f8-17103f55bf4a 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 40 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/1494126150,v1:172.21.15.153:6809/1494126150] [v2:172.21.15.153:6810/1494126150,v1:172.21.15.153:6811/1494126150] exists,up 9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 up in weight 1 up_from 48 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6818/1730932549,v1:172.21.15.78:6819/1730932549] [v2:172.21.15.78:6820/1730932549,v1:172.21.15.78:6821/1730932549] exists,up c4ef5595-a5ca-4f0e-82a7-f8322daab5e6 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 up in weight 1 up_from 53 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6816/999107021,v1:172.21.15.98:6817/999107021] [v2:172.21.15.98:6818/999107021,v1:172.21.15.98:6819/999107021] exists,up db466cbf-77b8-4cc4-95ca-a49e51ba27e6 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 up in weight 1 up_from 61 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/208983464,v1:172.21.15.153:6817/208983464] [v2:172.21.15.153:6818/208983464,v1:172.21.15.153:6819/208983464] exists,up e063f0ca-e609-4d53-8a22-f6ce8f3dae43 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 up in weight 1 up_from 69 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6826/2205607708,v1:172.21.15.78:6827/2205607708] [v2:172.21.15.78:6828/2205607708,v1:172.21.15.78:6829/2205607708] exists,up 66171eea-9d12-44e4-b449-b83fa2a1d52a 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 up in weight 1 up_from 75 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/652996874,v1:172.21.15.98:6825/652996874] [v2:172.21.15.98:6826/652996874,v1:172.21.15.98:6827/652996874] exists,up 72e85ff6-49af-4fc8-8114-9f33496e912b 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 up in weight 1 up_from 83 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/1068297198,v1:172.21.15.153:6825/1068297198] [v2:172.21.15.153:6826/1068297198,v1:172.21.15.153:6827/1068297198] exists,up e383aa05-dedb-4d76-9e7a-0411d04ef5ab 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.4 [2,11] 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.5 [8,11] 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.2 [3,9] 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.3 [4,9] 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.c [5,6] 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.17 [4,8] 2024-03-16T11:05:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.1d [4,7] 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.25 [5,9] 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.3a [4,8] 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/524003541 expires 2024-03-17T10:57:01.884008+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/2003716687 expires 2024-03-17T10:56:34.960962+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/3269664941 expires 2024-03-17T10:57:01.884008+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/2003716687 expires 2024-03-17T10:56:34.960962+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1201370297 expires 2024-03-17T10:56:34.960962+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2103714756 expires 2024-03-17T10:56:34.960962+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1410130144 expires 2024-03-17T10:57:01.884008+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/3269664941 expires 2024-03-17T10:57:01.884008+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2522710278 expires 2024-03-17T10:56:34.960962+0000 2024-03-16T11:05:44.007 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/4255056165 expires 2024-03-17T10:57:01.884008+0000 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59080a2890 msgr2=0x7f59080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59080a2890 0x7f59080a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5920103360 tx=0x7f591800b040 comp rx=0 tx=0).stop 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5920103500 msgr2=0x7f592019efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5920103500 0x7f592019efa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f59140036f0 tx=0x7f591400f710 comp rx=0 tx=0).stop 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 shutdown_connections 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59080a2890 0x7f59080a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5920102300 0x7f592019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5920103ea0 0x7f59201a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 --2- 172.21.15.78:0/2849333935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5920103500 0x7f592019efa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.005+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 >> 172.21.15.78:0/2849333935 conn(0x7f59200fd8e0 msgr2=0x7f592010e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:44.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.006+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 shutdown_connections 2024-03-16T11:05:44.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.006+0000 7f5924ab3700 1 -- 172.21.15.78:0/2849333935 wait complete. 2024-03-16T11:05:44.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2849333935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T11:05:44.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2849333935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T11:05:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2849333935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T11:05:44.645 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-16T11:05:44.657 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:44.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1898969413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8101910 msgr2=0x7f7fd8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:44.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1898969413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8101910 0x7f7fd8101d90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009900 tx=0x7f7fc002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1898969413 shutdown_connections 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1898969413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fd81022d0 0x7f7fd8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1898969413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8101910 0x7f7fd8101d90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1898969413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8100720 0x7f7fd8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1898969413 >> 172.21.15.78:0/1898969413 conn(0x7f7fd80fbc90 msgr2=0x7f7fd80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1898969413 shutdown_connections 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1898969413 wait complete. 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 Processor -- start 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.762+0000 7f7fddeb8640 1 -- start start 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fddeb8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8100720 0x7f7fd8195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fddeb8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fd8101910 0x7f7fd8195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fddeb8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd81022d0 0x7f7fd819cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fddeb8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd810b150 con 0x7f7fd8100720 2024-03-16T11:05:44.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fddeb8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd810afd0 con 0x7f7fd8101910 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fddeb8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd810b2d0 con 0x7f7fd81022d0 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8100720 0x7f7fd8195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8100720 0x7f7fd8195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50046/0 (socket says 172.21.15.78:50046) 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 -- 172.21.15.78:0/1368306743 learned_addr learned my addr 172.21.15.78:0/1368306743 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fceffd640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fd8101910 0x7f7fd8195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd7fff640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd81022d0 0x7f7fd819cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 -- 172.21.15.78:0/1368306743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd81022d0 msgr2=0x7f7fd819cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd81022d0 0x7f7fd819cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 -- 172.21.15.78:0/1368306743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fd8101910 msgr2=0x7f7fd8195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fd8101910 0x7f7fd8195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 -- 172.21.15.78:0/1368306743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd819d450 con 0x7f7fd8100720 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd77fe640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8100720 0x7f7fd8195830 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8010aa0 tx=0x7f7fc8010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:44.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.763+0000 7f7fd57fa640 1 -- 172.21.15.78:0/1368306743 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8004230 con 0x7f7fd8100720 2024-03-16T11:05:44.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.764+0000 7f7fd57fa640 1 -- 172.21.15.78:0/1368306743 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7fc800ee50 con 0x7f7fd8100720 2024-03-16T11:05:44.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.764+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd819d740 con 0x7f7fd8100720 2024-03-16T11:05:44.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.764+0000 7f7fd57fa640 1 -- 172.21.15.78:0/1368306743 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8016480 con 0x7f7fd8100720 2024-03-16T11:05:44.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.764+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd819dc10 con 0x7f7fd8100720 2024-03-16T11:05:44.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.765+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd8107b10 con 0x7f7fd8100720 2024-03-16T11:05:44.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.766+0000 7f7fd57fa640 1 -- 172.21.15.78:0/1368306743 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7fc8014050 con 0x7f7fd8100720 2024-03-16T11:05:44.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.766+0000 7f7fd57fa640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7fc40a2800 0x7f7fc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:44.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.766+0000 7f7fceffd640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7fc40a2800 0x7f7fc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:44.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.766+0000 7f7fd57fa640 1 -- 172.21.15.78:0/1368306743 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7fc80d16f0 con 0x7f7fd8100720 2024-03-16T11:05:44.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.767+0000 7f7fceffd640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7fc40a2800 0x7f7fc40a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009230 tx=0x7f7fc0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:44.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.770+0000 7f7fd57fa640 1 -- 172.21.15.78:0/1368306743 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7fc8012040 con 0x7f7fd8100720 2024-03-16T11:05:44.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.943+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7fd802d1b0 con 0x7f7fd8100720 2024-03-16T11:05:44.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.944+0000 7f7fd57fa640 1 -- 172.21.15.78:0/1368306743 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f7fc809e790 con 0x7f7fd8100720 2024-03-16T11:05:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:44.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.946+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7fc40a2800 msgr2=0x7f7fc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:44.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.946+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7fc40a2800 0x7f7fc40a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009230 tx=0x7f7fc0005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.946+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8100720 msgr2=0x7f7fd8195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.946+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8100720 0x7f7fd8195830 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8010aa0 tx=0x7f7fc8010ea0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 shutdown_connections 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7fc40a2800 0x7f7fc40a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd81022d0 0x7f7fd819cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fd8101910 0x7f7fd8195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 --2- 172.21.15.78:0/1368306743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fd8100720 0x7f7fd8195830 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 >> 172.21.15.78:0/1368306743 conn(0x7f7fd80fbc90 msgr2=0x7f7fd8104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 shutdown_connections 2024-03-16T11:05:44.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:44.947+0000 7f7fddeb8640 1 -- 172.21.15.78:0/1368306743 wait complete. 2024-03-16T11:05:44.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:05:44.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 -- 172.21.15.78:0/3319884494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 msgr2=0x7f5cac0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/3319884494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac0ffec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009930 tx=0x7f5ca802bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 -- 172.21.15.78:0/3319884494 shutdown_connections 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/3319884494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cac103430 0x7f5cac109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/3319884494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cac102a70 0x7f5cac102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/3319884494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac0ffec0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 -- 172.21.15.78:0/3319884494 >> 172.21.15.78:0/3319884494 conn(0x7f5cac0fb3d0 msgr2=0x7f5cac0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 -- 172.21.15.78:0/3319884494 shutdown_connections 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.064+0000 7f5cb35d4640 1 -- 172.21.15.78:0/3319884494 wait complete. 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 Processor -- start 2024-03-16T11:05:45.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 -- start start 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cac102a70 0x7f5cac197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cac103430 0x7f5cac19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5cac10d300 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5cac10d180 con 0x7f5cac103430 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5cac10d480 con 0x7f5cac102a70 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50064/0 (socket says 172.21.15.78:50064) 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 -- 172.21.15.78:0/2637011600 learned_addr learned my addr 172.21.15.78:0/2637011600 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1b71640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cac103430 0x7f5cac19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb0b6f640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cac102a70 0x7f5cac197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 -- 172.21.15.78:0/2637011600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cac102a70 msgr2=0x7f5cac197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cac102a70 0x7f5cac197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 -- 172.21.15.78:0/2637011600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cac103430 msgr2=0x7f5cac19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cac103430 0x7f5cac19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 -- 172.21.15.78:0/2637011600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cac19f6d0 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb1370640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac197ab0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8000c00 tx=0x7f5ca8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5ca27fc640 1 -- 172.21.15.78:0/2637011600 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8043020 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5ca27fc640 1 -- 172.21.15.78:0/2637011600 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ca8004aa0 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5ca27fc640 1 -- 172.21.15.78:0/2637011600 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8003c30 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.065+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cac19f960 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.066+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cac19ff00 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.067+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cac108d00 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.067+0000 7f5ca27fc640 1 -- 172.21.15.78:0/2637011600 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5ca8046020 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.068+0000 7f5ca27fc640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5c980a2880 0x7f5c980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.068+0000 7f5cb0b6f640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5c980a2880 0x7f5c980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.068+0000 7f5ca27fc640 1 -- 172.21.15.78:0/2637011600 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f5ca80f0e00 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.068+0000 7f5cb0b6f640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5c980a2880 0x7f5c980a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5cac198fc0 tx=0x7f5c9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.072+0000 7f5ca27fc640 1 -- 172.21.15.78:0/2637011600 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ca804d020 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.246+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5cac0623e0 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.247+0000 7f5ca27fc640 1 -- 172.21.15.78:0/2637011600 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f5ca80bdea0 con 0x7f5cac0ffaa0 2024-03-16T11:05:45.248 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5c980a2880 msgr2=0x7f5c980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5c980a2880 0x7f5c980a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5cac198fc0 tx=0x7f5c9c005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 msgr2=0x7f5cac197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac197ab0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8000c00 tx=0x7f5ca8004270 comp rx=0 tx=0).stop 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 shutdown_connections 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cac103430 0x7f5cac19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f5c980a2880 0x7f5c980a4d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cac102a70 0x7f5cac197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 --2- 172.21.15.78:0/2637011600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cac0ffaa0 0x7f5cac197ab0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 >> 172.21.15.78:0/2637011600 conn(0x7f5cac0fb3d0 msgr2=0x7f5cac105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:45.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 shutdown_connections 2024-03-16T11:05:45.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.249+0000 7f5cb35d4640 1 -- 172.21.15.78:0/2637011600 wait complete. 2024-03-16T11:05:45.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:45 smithi078 ceph-mon[32235]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:05:45.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:45 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1368306743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:45.260 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:05:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 -- 172.21.15.78:0/3671198654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1022a0 msgr2=0x7fc77c108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 --2- 172.21.15.78:0/3671198654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1022a0 0x7fc77c108b30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc774009ca0 tx=0x7fc77402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 -- 172.21.15.78:0/3671198654 shutdown_connections 2024-03-16T11:05:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 --2- 172.21.15.78:0/3671198654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1022a0 0x7fc77c108b30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 --2- 172.21.15.78:0/3671198654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc77c1018e0 0x7fc77c101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 --2- 172.21.15.78:0/3671198654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc77c1006f0 0x7fc77c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 -- 172.21.15.78:0/3671198654 >> 172.21.15.78:0/3671198654 conn(0x7fc77c0fbc80 msgr2=0x7fc77c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:45.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 -- 172.21.15.78:0/3671198654 shutdown_connections 2024-03-16T11:05:45.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.364+0000 7fc7823b9640 1 -- 172.21.15.78:0/3671198654 wait complete. 2024-03-16T11:05:45.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 Processor -- start 2024-03-16T11:05:45.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 -- start start 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc77c1006f0 0x7fc77c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1018e0 0x7fc77c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc77c1022a0 0x7fc77c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c10c220 con 0x7fc77c1018e0 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c10c0a0 con 0x7fc77c1006f0 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc7823b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c10c3a0 con 0x7fc77c1022a0 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1018e0 0x7fc77c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1018e0 0x7fc77c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50086/0 (socket says 172.21.15.78:50086) 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 -- 172.21.15.78:0/1616625116 learned_addr learned my addr 172.21.15.78:0/1616625116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc780956640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc77c1022a0 0x7fc77c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77bfff640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc77c1006f0 0x7fc77c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 -- 172.21.15.78:0/1616625116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc77c1022a0 msgr2=0x7fc77c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc77c1022a0 0x7fc77c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 -- 172.21.15.78:0/1616625116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc77c1006f0 msgr2=0x7fc77c1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc77c1006f0 0x7fc77c1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.365+0000 7fc77b7fe640 1 -- 172.21.15.78:0/1616625116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77c19f600 con 0x7fc77c1018e0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.366+0000 7fc77b7fe640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1018e0 0x7fc77c197f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc77000e9f0 tx=0x7fc77000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.366+0000 7fc7797fa640 1 -- 172.21.15.78:0/1616625116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77000cd80 con 0x7fc77c1018e0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.366+0000 7fc7797fa640 1 -- 172.21.15.78:0/1616625116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7700108e0 con 0x7fc77c1018e0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.366+0000 7fc7797fa640 1 -- 172.21.15.78:0/1616625116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770018920 con 0x7fc77c1018e0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.366+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc77c19f8f0 con 0x7fc77c1018e0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.366+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc77c19fe40 con 0x7fc77c1018e0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.367+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc77c1a0010 con 0x7fc77c1018e0 2024-03-16T11:05:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.368+0000 7fc7797fa640 1 -- 172.21.15.78:0/1616625116 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc770010a90 con 0x7fc77c1018e0 2024-03-16T11:05:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.368+0000 7fc7797fa640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7640a2800 0x7fc7640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.368+0000 7fc77bfff640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7640a2800 0x7fc7640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.368+0000 7fc7797fa640 1 -- 172.21.15.78:0/1616625116 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc7700cf220 con 0x7fc77c1018e0 2024-03-16T11:05:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.369+0000 7fc77bfff640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7640a2800 0x7fc7640a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc77c101740 tx=0x7fc76c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.372+0000 7fc7797fa640 1 -- 172.21.15.78:0/1616625116 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc77009bc70 con 0x7fc77c1018e0 2024-03-16T11:05:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:45 smithi098 ceph-mon[35848]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:05:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:45 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1368306743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:45 smithi153 ceph-mon[35521]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:05:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:45 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1368306743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:45.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.546+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc77c1a03a0 con 0x7fc77c1018e0 2024-03-16T11:05:45.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.546+0000 7fc7797fa640 1 -- 172.21.15.78:0/1616625116 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fc77009fd10 con 0x7fc77c1018e0 2024-03-16T11:05:45.546 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:05:45.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7640a2800 msgr2=0x7fc7640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7640a2800 0x7fc7640a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc77c101740 tx=0x7fc76c005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1018e0 msgr2=0x7fc77c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1018e0 0x7fc77c197f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc77000e9f0 tx=0x7fc77000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 shutdown_connections 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc7640a2800 0x7fc7640a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc77c1022a0 0x7fc77c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc77c1018e0 0x7fc77c197f20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 --2- 172.21.15.78:0/1616625116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc77c1006f0 0x7fc77c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 >> 172.21.15.78:0/1616625116 conn(0x7fc77c0fbc80 msgr2=0x7fc77c0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 shutdown_connections 2024-03-16T11:05:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.548+0000 7fc7823b9640 1 -- 172.21.15.78:0/1616625116 wait complete. 2024-03-16T11:05:45.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:05:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 -- 172.21.15.78:0/3102197826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc101810 msgr2=0x7fe3bc101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/3102197826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc101810 0x7fe3bc101c30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac009ca0 tx=0x7fe3ac02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 -- 172.21.15.78:0/3102197826 shutdown_connections 2024-03-16T11:05:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/3102197826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3bc1033c0 0x7fe3bc109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/3102197826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc102a00 0x7fe3bc102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/3102197826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc101810 0x7fe3bc101c30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 -- 172.21.15.78:0/3102197826 >> 172.21.15.78:0/3102197826 conn(0x7fe3bc0fcd80 msgr2=0x7fe3bc0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.663+0000 7fe3c0d97640 1 -- 172.21.15.78:0/3102197826 shutdown_connections 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 -- 172.21.15.78:0/3102197826 wait complete. 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 Processor -- start 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 -- start start 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3bc101810 0x7fe3bc075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102a00 0x7fe3bc073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc1033c0 0x7fe3bc074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc10d330 con 0x7fe3bc102a00 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc10d1b0 con 0x7fe3bc101810 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3c0d97640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc10d4b0 con 0x7fe3bc1033c0 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3b9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102a00 0x7fe3bc073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3b9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102a00 0x7fe3bc073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35850/0 (socket says 172.21.15.78:35850) 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.664+0000 7fe3b9d9b640 1 -- 172.21.15.78:0/2778478122 learned_addr learned my addr 172.21.15.78:0/2778478122 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3bad9d640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc1033c0 0x7fe3bc074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3ba59c640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3bc101810 0x7fe3bc075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3b9d9b640 1 -- 172.21.15.78:0/2778478122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc1033c0 msgr2=0x7fe3bc074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3b9d9b640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc1033c0 0x7fe3bc074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3b9d9b640 1 -- 172.21.15.78:0/2778478122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3bc101810 msgr2=0x7fe3bc075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3b9d9b640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3bc101810 0x7fe3bc075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3b9d9b640 1 -- 172.21.15.78:0/2778478122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3bc0749a0 con 0x7fe3bc102a00 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3b9d9b640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102a00 0x7fe3bc073bd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9f0 tx=0x7fe3b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/2778478122 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000ccc0 con 0x7fe3bc102a00 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/2778478122 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3b000ce60 con 0x7fe3bc102a00 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/2778478122 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0010430 con 0x7fe3bc102a00 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3bc1a1c00 con 0x7fe3bc102a00 2024-03-16T11:05:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.665+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3bc1a2100 con 0x7fe3bc102a00 2024-03-16T11:05:45.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.666+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3bc108c50 con 0x7fe3bc102a00 2024-03-16T11:05:45.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.667+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/2778478122 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe3b0005000 con 0x7fe3bc102a00 2024-03-16T11:05:45.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.668+0000 7fe3ab7fe640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe3a40a2800 0x7fe3a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.668+0000 7fe3ba59c640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe3a40a2800 0x7fe3a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.668+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/2778478122 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe3b00cefc0 con 0x7fe3bc102a00 2024-03-16T11:05:45.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.668+0000 7fe3ba59c640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe3a40a2800 0x7fe3a40a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac032380 tx=0x7fe3ac000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.671+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/2778478122 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3b0053740 con 0x7fe3bc102a00 2024-03-16T11:05:45.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.844+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe3bc02d3b0 con 0x7fe3bc102a00 2024-03-16T11:05:45.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.845+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/2778478122 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fe3b009c100 con 0x7fe3bc102a00 2024-03-16T11:05:45.845 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:05:45.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.846+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe3a40a2800 msgr2=0x7fe3a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.846+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe3a40a2800 0x7fe3a40a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac032380 tx=0x7fe3ac000f40 comp rx=0 tx=0).stop 2024-03-16T11:05:45.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.846+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102a00 msgr2=0x7fe3bc073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.846+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102a00 0x7fe3bc073bd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9f0 tx=0x7fe3b000edf0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 shutdown_connections 2024-03-16T11:05:45.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe3a40a2800 0x7fe3a40a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc1033c0 0x7fe3bc074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102a00 0x7fe3bc073bd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 --2- 172.21.15.78:0/2778478122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3bc101810 0x7fe3bc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 >> 172.21.15.78:0/2778478122 conn(0x7fe3bc0fcd80 msgr2=0x7fe3bc105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:45.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 shutdown_connections 2024-03-16T11:05:45.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.847+0000 7fe3c0d97640 1 -- 172.21.15.78:0/2778478122 wait complete. 2024-03-16T11:05:45.858 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:05:45.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 -- 172.21.15.78:0/720124823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e00102ac0 msgr2=0x7f9e00102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 --2- 172.21.15.78:0/720124823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e00102ac0 0x7f9e00102f40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009930 tx=0x7f9df402bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 -- 172.21.15.78:0/720124823 shutdown_connections 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 --2- 172.21.15.78:0/720124823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e00103480 0x7f9e00109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 --2- 172.21.15.78:0/720124823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e00102ac0 0x7f9e00102f40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 --2- 172.21.15.78:0/720124823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e000ffaf0 0x7f9e000fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 -- 172.21.15.78:0/720124823 >> 172.21.15.78:0/720124823 conn(0x7f9e000fb420 msgr2=0x7f9e000fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 -- 172.21.15.78:0/720124823 shutdown_connections 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.961+0000 7f9e04d09640 1 -- 172.21.15.78:0/720124823 wait complete. 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 Processor -- start 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 -- start start 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e000ffaf0 0x7f9e00197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e00102ac0 0x7f9e00197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e00103480 0x7f9e0019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0010d380 con 0x7f9e000ffaf0 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0010d200 con 0x7f9e00102ac0 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9e04d09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0010d500 con 0x7f9e00103480 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e000ffaf0 0x7f9e00197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e000ffaf0 0x7f9e00197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35876/0 (socket says 172.21.15.78:35876) 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 -- 172.21.15.78:0/746257135 learned_addr learned my addr 172.21.15.78:0/746257135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfed9d640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e00103480 0x7f9e0019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfdd9b640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e00102ac0 0x7f9e00197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 -- 172.21.15.78:0/746257135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e00103480 msgr2=0x7f9e0019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e00103480 0x7f9e0019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 -- 172.21.15.78:0/746257135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e00102ac0 msgr2=0x7f9e00197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e00102ac0 0x7f9e00197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:45.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 -- 172.21.15.78:0/746257135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0019f6d0 con 0x7f9e000ffaf0 2024-03-16T11:05:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.962+0000 7f9dfe59c640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e000ffaf0 0x7f9e00197ab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e9c0 tx=0x7f9df000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.963+0000 7f9def7fe640 1 -- 172.21.15.78:0/746257135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df000cd80 con 0x7f9e000ffaf0 2024-03-16T11:05:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.963+0000 7f9def7fe640 1 -- 172.21.15.78:0/746257135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9df00035f0 con 0x7f9e000ffaf0 2024-03-16T11:05:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.963+0000 7f9def7fe640 1 -- 172.21.15.78:0/746257135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0010690 con 0x7f9e000ffaf0 2024-03-16T11:05:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.963+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0019f9c0 con 0x7f9e000ffaf0 2024-03-16T11:05:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.963+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0019fee0 con 0x7f9e000ffaf0 2024-03-16T11:05:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.964+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e00108d60 con 0x7f9e000ffaf0 2024-03-16T11:05:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.965+0000 7f9def7fe640 1 -- 172.21.15.78:0/746257135 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9df00037a0 con 0x7f9e000ffaf0 2024-03-16T11:05:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.965+0000 7f9def7fe640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9de80a2880 0x7f9de80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.966+0000 7f9dfdd9b640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9de80a2880 0x7f9de80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.966+0000 7f9def7fe640 1 -- 172.21.15.78:0/746257135 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9df00ceff0 con 0x7f9e000ffaf0 2024-03-16T11:05:45.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.966+0000 7f9dfdd9b640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9de80a2880 0x7f9de80a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009900 tx=0x7f9df4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:45.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:45.969+0000 7f9def7fe640 1 -- 172.21.15.78:0/746257135 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9df009c130 con 0x7f9e000ffaf0 2024-03-16T11:05:46.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.187+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e0002d1e0 con 0x7f9e000ffaf0 2024-03-16T11:05:46.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.188+0000 7f9def7fe640 1 -- 172.21.15.78:0/746257135 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11869 (secure 0 0 0) 0x7f9df009ba30 con 0x7f9e000ffaf0 2024-03-16T11:05:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:46.188 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:04:03.658334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24488,24505,24521]},"id":1}]} 2024-03-16T11:05:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 21 2024-03-16T11:05:46.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9de80a2880 msgr2=0x7f9de80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9de80a2880 0x7f9de80a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009900 tx=0x7f9df4005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e000ffaf0 msgr2=0x7f9e00197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e000ffaf0 0x7f9e00197ab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e9c0 tx=0x7f9df000ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 shutdown_connections 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9de80a2880 0x7f9de80a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e00103480 0x7f9e0019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e00102ac0 0x7f9e00197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 --2- 172.21.15.78:0/746257135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e000ffaf0 0x7f9e00197ab0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 >> 172.21.15.78:0/746257135 conn(0x7f9e000fb420 msgr2=0x7f9e00105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 shutdown_connections 2024-03-16T11:05:46.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.190+0000 7f9e04d09640 1 -- 172.21.15.78:0/746257135 wait complete. 2024-03-16T11:05:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:46 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2637011600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:46 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1616625116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:46 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2778478122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3683757503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea01022e0 msgr2=0x7f8ea01046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3683757503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea01022e0 0x7f8ea01046d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009900 tx=0x7f8e9002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3683757503 shutdown_connections 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3683757503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea0107540 0x7f8ea0109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3683757503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea0104c10 0x7f8ea0107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3683757503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea01022e0 0x7f8ea01046d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3683757503 >> 172.21.15.78:0/3683757503 conn(0x7f8ea00fbc70 msgr2=0x7f8ea00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3683757503 shutdown_connections 2024-03-16T11:05:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.303+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3683757503 wait complete. 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 Processor -- start 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 -- start start 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea01022e0 0x7f8ea0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea0104c10 0x7f8ea0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea0107540 0x7f8ea019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea010d350 con 0x7f8ea0107540 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea010d1d0 con 0x7f8ea0104c10 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea6cc6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea010d4d0 con 0x7f8ea01022e0 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea0107540 0x7f8ea019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea0107540 0x7f8ea019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35894/0 (socket says 172.21.15.78:35894) 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 -- 172.21.15.78:0/3550754319 learned_addr learned my addr 172.21.15.78:0/3550754319 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea4a62640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea01022e0 0x7f8ea0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8e9ffff640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea0104c10 0x7f8ea0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 -- 172.21.15.78:0/3550754319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea01022e0 msgr2=0x7f8ea0197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea01022e0 0x7f8ea0197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 -- 172.21.15.78:0/3550754319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea0104c10 msgr2=0x7f8ea0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea0104c10 0x7f8ea0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.304+0000 7f8ea5263640 1 -- 172.21.15.78:0/3550754319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea019f680 con 0x7f8ea0107540 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.305+0000 7f8ea5263640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea0107540 0x7f8ea019f010 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800eb10 tx=0x7f8e9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.305+0000 7f8e9dffb640 1 -- 172.21.15.78:0/3550754319 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98004b20 con 0x7f8ea0107540 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.305+0000 7f8e9dffb640 1 -- 172.21.15.78:0/3550754319 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e98004cc0 con 0x7f8ea0107540 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.305+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea019f970 con 0x7f8ea0107540 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.305+0000 7f8e9dffb640 1 -- 172.21.15.78:0/3550754319 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98010690 con 0x7f8ea0107540 2024-03-16T11:05:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.305+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea019fec0 con 0x7f8ea0107540 2024-03-16T11:05:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.306+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea00fd760 con 0x7f8ea0107540 2024-03-16T11:05:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.307+0000 7f8e9dffb640 1 -- 172.21.15.78:0/3550754319 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8e98005020 con 0x7f8ea0107540 2024-03-16T11:05:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.307+0000 7f8e9dffb640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e880a2800 0x7f8e880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.307+0000 7f8ea4a62640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e880a2800 0x7f8e880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.307+0000 7f8e9dffb640 1 -- 172.21.15.78:0/3550754319 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8e980cf120 con 0x7f8ea0107540 2024-03-16T11:05:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.308+0000 7f8ea4a62640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e880a2800 0x7f8e880a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009230 tx=0x7f8e90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.312+0000 7f8e9dffb640 1 -- 172.21.15.78:0/3550754319 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e9809c1c0 con 0x7f8ea0107540 2024-03-16T11:05:46.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:46 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2637011600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:46 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1616625116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:46 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2778478122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.490+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ea00623e0 con 0x7f8ea0107540 2024-03-16T11:05:46.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.492+0000 7f8e9dffb640 1 -- 172.21.15.78:0/3550754319 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f8e9809bac0 con 0x7f8ea0107540 2024-03-16T11:05:46.493 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:46.493 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:46.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e880a2800 msgr2=0x7f8e880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e880a2800 0x7f8e880a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009230 tx=0x7f8e90005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea0107540 msgr2=0x7f8ea019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea0107540 0x7f8ea019f010 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800eb10 tx=0x7f8e9800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 shutdown_connections 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea0107540 0x7f8ea019f010 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea0104c10 0x7f8ea0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e880a2800 0x7f8e880a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 --2- 172.21.15.78:0/3550754319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea01022e0 0x7f8ea0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 >> 172.21.15.78:0/3550754319 conn(0x7f8ea00fbc70 msgr2=0x7f8ea0102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 shutdown_connections 2024-03-16T11:05:46.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.494+0000 7f8ea6cc6640 1 -- 172.21.15.78:0/3550754319 wait complete. 2024-03-16T11:05:46.506 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:05:46.507 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:05:46.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:46 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2637011600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:46 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1616625116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:46 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2778478122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:46.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 -- 172.21.15.78:0/3807392592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0101910 msgr2=0x7f38b0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 --2- 172.21.15.78:0/3807392592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0101910 0x7f38b0101d90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009860 tx=0x7f38a402bb70 comp rx=0 tx=0).stop 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 -- 172.21.15.78:0/3807392592 shutdown_connections 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 --2- 172.21.15.78:0/3807392592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b01022d0 0x7f38b0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 --2- 172.21.15.78:0/3807392592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0101910 0x7f38b0101d90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 --2- 172.21.15.78:0/3807392592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 0x7f38b0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 -- 172.21.15.78:0/3807392592 >> 172.21.15.78:0/3807392592 conn(0x7f38b00fbc90 msgr2=0x7f38b00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 -- 172.21.15.78:0/3807392592 shutdown_connections 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.611+0000 7f38b6612640 1 -- 172.21.15.78:0/3807392592 wait complete. 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 Processor -- start 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 -- start start 2024-03-16T11:05:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 0x7f38b0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b0101910 0x7f38b0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b01022d0 0x7f38b019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010d370 con 0x7f38b0100720 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010d1f0 con 0x7f38b0101910 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b6612640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010d4f0 con 0x7f38b01022d0 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 0x7f38b0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 0x7f38b0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35910/0 (socket says 172.21.15.78:35910) 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 -- 172.21.15.78:0/3837112866 learned_addr learned my addr 172.21.15.78:0/3837112866 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38b4baf640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b01022d0 0x7f38b019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38af7fe640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b0101910 0x7f38b0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 -- 172.21.15.78:0/3837112866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b01022d0 msgr2=0x7f38b019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b01022d0 0x7f38b019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 -- 172.21.15.78:0/3837112866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b0101910 msgr2=0x7f38b0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b0101910 0x7f38b0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 -- 172.21.15.78:0/3837112866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b019f660 con 0x7f38b0100720 2024-03-16T11:05:46.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.612+0000 7f38affff640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 0x7f38b0197a90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f38a000e9f0 tx=0x7f38a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:46.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.613+0000 7f38ad7fa640 1 -- 172.21.15.78:0/3837112866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a000ccc0 con 0x7f38b0100720 2024-03-16T11:05:46.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.613+0000 7f38ad7fa640 1 -- 172.21.15.78:0/3837112866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38a00035f0 con 0x7f38b0100720 2024-03-16T11:05:46.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.613+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b019f950 con 0x7f38b0100720 2024-03-16T11:05:46.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.613+0000 7f38ad7fa640 1 -- 172.21.15.78:0/3837112866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a000ce60 con 0x7f38b0100720 2024-03-16T11:05:46.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.613+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b019fe70 con 0x7f38b0100720 2024-03-16T11:05:46.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.614+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b0107c00 con 0x7f38b0100720 2024-03-16T11:05:46.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.614+0000 7f38ad7fa640 1 -- 172.21.15.78:0/3837112866 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f38a00037a0 con 0x7f38b0100720 2024-03-16T11:05:46.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.615+0000 7f38ad7fa640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38980a2800 0x7f38980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.615+0000 7f38af7fe640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38980a2800 0x7f38980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.615+0000 7f38ad7fa640 1 -- 172.21.15.78:0/3837112866 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f38a00cecb0 con 0x7f38b0100720 2024-03-16T11:05:46.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.615+0000 7f38af7fe640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38980a2800 0x7f38980a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000c00 tx=0x7f38a40349a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:46.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.619+0000 7f38ad7fa640 1 -- 172.21.15.78:0/3837112866 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f38a009bd50 con 0x7f38b0100720 2024-03-16T11:05:46.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.792+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38b00623e0 con 0x7f38b0100720 2024-03-16T11:05:46.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.792+0000 7f38ad7fa640 1 -- 172.21.15.78:0/3837112866 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f38a009b650 con 0x7f38b0100720 2024-03-16T11:05:46.793 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38980a2800 msgr2=0x7f38980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38980a2800 0x7f38980a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000c00 tx=0x7f38a40349a0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 msgr2=0x7f38b0197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 0x7f38b0197a90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f38a000e9f0 tx=0x7f38a000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 shutdown_connections 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38980a2800 0x7f38980a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b01022d0 0x7f38b019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b0101910 0x7f38b0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 --2- 172.21.15.78:0/3837112866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38b0100720 0x7f38b0197a90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 >> 172.21.15.78:0/3837112866 conn(0x7f38b00fbc90 msgr2=0x7f38b0104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:46.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 shutdown_connections 2024-03-16T11:05:46.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.794+0000 7f38b6612640 1 -- 172.21.15.78:0/3837112866 wait complete. 2024-03-16T11:05:46.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:05:46.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.910+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/2422202267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 msgr2=0x7fdfd8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.910+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/2422202267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8101c30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009900 tx=0x7fdfcc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/2422202267 shutdown_connections 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/2422202267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd8102170 0x7fdfd8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/2422202267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8101c30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/2422202267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81005c0 0x7fdfd81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/2422202267 >> 172.21.15.78:0/2422202267 conn(0x7fdfd80fbb50 msgr2=0x7fdfd80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/2422202267 shutdown_connections 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/2422202267 wait complete. 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 Processor -- start 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.911+0000 7fdfdc9fe640 1 -- start start 2024-03-16T11:05:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81005c0 0x7fdfd81978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd8102170 0x7fdfd819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810d210 con 0x7fdfd81017b0 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810d090 con 0x7fdfd81005c0 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810d390 con 0x7fdfd8102170 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35922/0 (socket says 172.21.15.78:35922) 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 -- 172.21.15.78:0/1765278114 learned_addr learned my addr 172.21.15.78:0/1765278114 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd6d9d640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81005c0 0x7fdfd81978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd759e640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd8102170 0x7fdfd819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 -- 172.21.15.78:0/1765278114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd8102170 msgr2=0x7fdfd819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd8102170 0x7fdfd819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 -- 172.21.15.78:0/1765278114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81005c0 msgr2=0x7fdfd81978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81005c0 0x7fdfd81978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 -- 172.21.15.78:0/1765278114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd819f4d0 con 0x7fdfd81017b0 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfd659c640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8197df0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006010 tx=0x7fdfcc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfbffff640 1 -- 172.21.15.78:0/1765278114 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc043020 con 0x7fdfd81017b0 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfbffff640 1 -- 172.21.15.78:0/1765278114 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfcc0043d0 con 0x7fdfd81017b0 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd819f760 con 0x7fdfd81017b0 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfbffff640 1 -- 172.21.15.78:0/1765278114 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc003b80 con 0x7fdfd81017b0 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.912+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd819fcb0 con 0x7fdfd81017b0 2024-03-16T11:05:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.913+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd81079b0 con 0x7fdfd81017b0 2024-03-16T11:05:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.915+0000 7fdfbffff640 1 -- 172.21.15.78:0/1765278114 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdfcc046020 con 0x7fdfd81017b0 2024-03-16T11:05:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.915+0000 7fdfbffff640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2800 0x7fdfc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.915+0000 7fdfd6d9d640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2800 0x7fdfc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.915+0000 7fdfbffff640 1 -- 172.21.15.78:0/1765278114 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdfcc0f1270 con 0x7fdfd81017b0 2024-03-16T11:05:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.916+0000 7fdfd6d9d640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2800 0x7fdfc00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8101610 tx=0x7fdfc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:46.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:46.919+0000 7fdfbffff640 1 -- 172.21.15.78:0/1765278114 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdfcc04d020 con 0x7fdfd81017b0 2024-03-16T11:05:47.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.097+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdfd802d340 con 0x7fdfd81017b0 2024-03-16T11:05:47.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.097+0000 7fdfbffff640 1 -- 172.21.15.78:0/1765278114 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fdfcc0be310 con 0x7fdfd81017b0 2024-03-16T11:05:47.098 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2800 msgr2=0x7fdfc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2800 0x7fdfc00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8101610 tx=0x7fdfc8005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 msgr2=0x7fdfd8197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8197df0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006010 tx=0x7fdfcc005e40 comp rx=0 tx=0).stop 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 shutdown_connections 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdfc00a2800 0x7fdfc00a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd8102170 0x7fdfd819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd81017b0 0x7fdfd8197df0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 --2- 172.21.15.78:0/1765278114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfd81005c0 0x7fdfd81978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 >> 172.21.15.78:0/1765278114 conn(0x7fdfd80fbb50 msgr2=0x7fdfd8104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:47.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 shutdown_connections 2024-03-16T11:05:47.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.099+0000 7fdfdc9fe640 1 -- 172.21.15.78:0/1765278114 wait complete. 2024-03-16T11:05:47.111 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:05:47.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 -- 172.21.15.78:0/1539876452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 msgr2=0x7f3890101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 --2- 172.21.15.78:0/1539876452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890101d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3884009ca0 tx=0x7f388402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 -- 172.21.15.78:0/1539876452 shutdown_connections 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 --2- 172.21.15.78:0/1539876452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38901022a0 0x7f3890108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 --2- 172.21.15.78:0/1539876452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890101d60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 --2- 172.21.15.78:0/1539876452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38901006f0 0x7f3890100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 -- 172.21.15.78:0/1539876452 >> 172.21.15.78:0/1539876452 conn(0x7f38900fbc80 msgr2=0x7f38900fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 -- 172.21.15.78:0/1539876452 shutdown_connections 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 -- 172.21.15.78:0/1539876452 wait complete. 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 Processor -- start 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.214+0000 7f3894861640 1 -- start start 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38901006f0 0x7f3890197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38901022a0 0x7f389019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f389010c220 con 0x7f38901018e0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f389010c0a0 con 0x7f38901022a0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f389010c3a0 con 0x7f38901006f0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35938/0 (socket says 172.21.15.78:35938) 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 -- 172.21.15.78:0/2309683328 learned_addr learned my addr 172.21.15.78:0/2309683328 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388e59c640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38901006f0 0x7f3890197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388ed9d640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38901022a0 0x7f389019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 -- 172.21.15.78:0/2309683328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38901006f0 msgr2=0x7f3890197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38901006f0 0x7f3890197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 -- 172.21.15.78:0/2309683328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38901022a0 msgr2=0x7f389019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38901022a0 0x7f389019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 -- 172.21.15.78:0/2309683328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389019f650 con 0x7f38901018e0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f388dd9b640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890197f70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3884009dc0 tx=0x7f38840328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:47.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f387f7fe640 1 -- 172.21.15.78:0/2309683328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3884041020 con 0x7f38901018e0 2024-03-16T11:05:47.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f387f7fe640 1 -- 172.21.15.78:0/2309683328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3884003e20 con 0x7f38901018e0 2024-03-16T11:05:47.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f387f7fe640 1 -- 172.21.15.78:0/2309683328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3884039040 con 0x7f38901018e0 2024-03-16T11:05:47.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389019f8e0 con 0x7f38901018e0 2024-03-16T11:05:47.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.215+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389019fe20 con 0x7f38901018e0 2024-03-16T11:05:47.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.216+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38901a0050 con 0x7f38901018e0 2024-03-16T11:05:47.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.217+0000 7f387f7fe640 1 -- 172.21.15.78:0/2309683328 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3884044020 con 0x7f38901018e0 2024-03-16T11:05:47.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.218+0000 7f387f7fe640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38780a2880 0x7f38780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.218+0000 7f388e59c640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38780a2880 0x7f38780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.218+0000 7f387f7fe640 1 -- 172.21.15.78:0/2309683328 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f38840f0ae0 con 0x7f38901018e0 2024-03-16T11:05:47.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.218+0000 7f388e59c640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38780a2880 0x7f38780a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3890101740 tx=0x7f3888005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.223+0000 7f387f7fe640 1 -- 172.21.15.78:0/2309683328 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f388404b020 con 0x7f38901018e0 2024-03-16T11:05:47.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.396+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f38901a0220 con 0x7f38901018e0 2024-03-16T11:05:47.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.396+0000 7f387f7fe640 1 -- 172.21.15.78:0/2309683328 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f38840bdb80 con 0x7f38901018e0 2024-03-16T11:05:47.397 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38780a2880 msgr2=0x7f38780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38780a2880 0x7f38780a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3890101740 tx=0x7f3888005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 msgr2=0x7f3890197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890197f70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3884009dc0 tx=0x7f38840328a0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 shutdown_connections 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38901022a0 0x7f389019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38901018e0 0x7f3890197f70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f38780a2880 0x7f38780a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 --2- 172.21.15.78:0/2309683328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38901006f0 0x7f3890197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 >> 172.21.15.78:0/2309683328 conn(0x7f38900fbc80 msgr2=0x7f3890069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:47.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 shutdown_connections 2024-03-16T11:05:47.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.398+0000 7f3894861640 1 -- 172.21.15.78:0/2309683328 wait complete. 2024-03-16T11:05:47.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:47 smithi078 ceph-mon[32235]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T11:05:47.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:47 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/746257135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:47.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:47 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3550754319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:47.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:47 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3837112866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:47.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:05:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:47 smithi098 ceph-mon[35848]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T11:05:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:47 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/746257135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:47 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3550754319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:47 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3837112866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 -- 172.21.15.78:0/3223375991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c104c30 msgr2=0x7f898c107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 --2- 172.21.15.78:0/3223375991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c104c30 0x7f898c107020 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f897c009900 tx=0x7f897c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 -- 172.21.15.78:0/3223375991 shutdown_connections 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 --2- 172.21.15.78:0/3223375991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f898c107560 0x7f898c1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 --2- 172.21.15.78:0/3223375991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c104c30 0x7f898c107020 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 --2- 172.21.15.78:0/3223375991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898c102300 0x7f898c1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 -- 172.21.15.78:0/3223375991 >> 172.21.15.78:0/3223375991 conn(0x7f898c0fbd10 msgr2=0x7f898c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:47.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 -- 172.21.15.78:0/3223375991 shutdown_connections 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.511+0000 7f8993f97640 1 -- 172.21.15.78:0/3223375991 wait complete. 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 Processor -- start 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 -- start start 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f898c102300 0x7f898c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898c104c30 0x7f898c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c107560 0x7f898c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f898c10d350 con 0x7f898c107560 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f898c10d1d0 con 0x7f898c102300 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8993f97640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f898c10d4d0 con 0x7f898c104c30 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c107560 0x7f898c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c107560 0x7f898c19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35956/0 (socket says 172.21.15.78:35956) 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 -- 172.21.15.78:0/3654281853 learned_addr learned my addr 172.21.15.78:0/3654281853 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8991d33640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f898c102300 0x7f898c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8991532640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898c104c30 0x7f898c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 -- 172.21.15.78:0/3654281853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898c104c30 msgr2=0x7f898c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898c104c30 0x7f898c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 -- 172.21.15.78:0/3654281853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f898c102300 msgr2=0x7f898c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f898c102300 0x7f898c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 -- 172.21.15.78:0/3654281853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c19f6a0 con 0x7f898c107560 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8991d33640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f898c102300 0x7f898c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.512+0000 7f8992534640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c107560 0x7f898c19f030 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f898400eb10 tx=0x7f898400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.513+0000 7f8982ffd640 1 -- 172.21.15.78:0/3654281853 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984004b20 con 0x7f898c107560 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.513+0000 7f8982ffd640 1 -- 172.21.15.78:0/3654281853 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8984004cc0 con 0x7f898c107560 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.513+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898c19f990 con 0x7f898c107560 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.513+0000 7f8982ffd640 1 -- 172.21.15.78:0/3654281853 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984010690 con 0x7f898c107560 2024-03-16T11:05:47.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.513+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f898c19feb0 con 0x7f898c107560 2024-03-16T11:05:47.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.514+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f898c0fd800 con 0x7f898c107560 2024-03-16T11:05:47.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.515+0000 7f8982ffd640 1 -- 172.21.15.78:0/3654281853 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8984005000 con 0x7f898c107560 2024-03-16T11:05:47.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.516+0000 7f8982ffd640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f89780a2880 0x7f89780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.516+0000 7f8991d33640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f89780a2880 0x7f89780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.516+0000 7f8982ffd640 1 -- 172.21.15.78:0/3654281853 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f89840cf190 con 0x7f898c107560 2024-03-16T11:05:47.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.516+0000 7f8991d33640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f89780a2880 0x7f89780a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8988009fc0 tx=0x7f8988005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:47.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.520+0000 7f8982ffd640 1 -- 172.21.15.78:0/3654281853 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8984053740 con 0x7f898c107560 2024-03-16T11:05:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:47 smithi153 ceph-mon[35521]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T11:05:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:47 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/746257135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:47 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3550754319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:47 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3837112866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:47.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.733+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f898c02d1b0 con 0x7f898c107560 2024-03-16T11:05:47.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.733+0000 7f8982ffd640 1 -- 172.21.15.78:0/3654281853 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11869 (secure 0 0 0) 0x7f898409c1f0 con 0x7f898c107560 2024-03-16T11:05:47.734 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:47.734 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:04:03.658334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24488,24505,24521]},"id":1}]} 2024-03-16T11:05:47.735 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 21 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f89780a2880 msgr2=0x7f89780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f89780a2880 0x7f89780a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8988009fc0 tx=0x7f8988005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c107560 msgr2=0x7f898c19f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c107560 0x7f898c19f030 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f898400eb10 tx=0x7f898400c2d0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 shutdown_connections 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f898c107560 0x7f898c19f030 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f89780a2880 0x7f89780a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898c104c30 0x7f898c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 --2- 172.21.15.78:0/3654281853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f898c102300 0x7f898c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 >> 172.21.15.78:0/3654281853 conn(0x7f898c0fbd10 msgr2=0x7f898c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:47.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 shutdown_connections 2024-03-16T11:05:47.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.735+0000 7f8993f97640 1 -- 172.21.15.78:0/3654281853 wait complete. 2024-03-16T11:05:47.747 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:47.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 -- 172.21.15.78:0/1899604616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1022b0 msgr2=0x7f6d9c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1899604616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1022b0 0x7f6d9c108b40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009ca0 tx=0x7f6d8c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 -- 172.21.15.78:0/1899604616 shutdown_connections 2024-03-16T11:05:47.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1899604616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1022b0 0x7f6d9c108b40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1899604616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c1018f0 0x7f6d9c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1899604616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 0x7f6d9c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 -- 172.21.15.78:0/1899604616 >> 172.21.15.78:0/1899604616 conn(0x7f6d9c0fbc90 msgr2=0x7f6d9c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 -- 172.21.15.78:0/1899604616 shutdown_connections 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 -- 172.21.15.78:0/1899604616 wait complete. 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 Processor -- start 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 -- start start 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.848+0000 7f6da18bf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 0x7f6d9c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da18bf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1018f0 0x7f6d9c196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da18bf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c1022b0 0x7f6d9c19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da18bf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10c230 con 0x7f6d9c100700 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da18bf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10c0b0 con 0x7f6d9c1018f0 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da18bf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10c3b0 con 0x7f6d9c1022b0 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 0x7f6d9c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 0x7f6d9c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35974/0 (socket says 172.21.15.78:35974) 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 -- 172.21.15.78:0/1393795070 learned_addr learned my addr 172.21.15.78:0/1393795070 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9a7fc640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1018f0 0x7f6d9c196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9b7fe640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c1022b0 0x7f6d9c19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 -- 172.21.15.78:0/1393795070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c1022b0 msgr2=0x7f6d9c19de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c1022b0 0x7f6d9c19de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 -- 172.21.15.78:0/1393795070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1018f0 msgr2=0x7f6d9c196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1018f0 0x7f6d9c196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 -- 172.21.15.78:0/1393795070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c19e4f0 con 0x7f6d9c100700 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6d9affd640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 0x7f6d9c1968d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400e9c0 tx=0x7f6d9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da08bd640 1 -- 172.21.15.78:0/1393795070 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9400cd80 con 0x7f6d9c100700 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da08bd640 1 -- 172.21.15.78:0/1393795070 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d940035f0 con 0x7f6d9c100700 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9c19e7e0 con 0x7f6d9c100700 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da08bd640 1 -- 172.21.15.78:0/1393795070 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94010690 con 0x7f6d9c100700 2024-03-16T11:05:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.849+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9c19ed80 con 0x7f6d9c100700 2024-03-16T11:05:47.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.850+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d9c107a20 con 0x7f6d9c100700 2024-03-16T11:05:47.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.851+0000 7f6da08bd640 1 -- 172.21.15.78:0/1393795070 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6d940037a0 con 0x7f6d9c100700 2024-03-16T11:05:47.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.852+0000 7f6da08bd640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6d880a2880 0x7f6d880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:47.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.852+0000 7f6da08bd640 1 -- 172.21.15.78:0/1393795070 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f6d940ced70 con 0x7f6d9c100700 2024-03-16T11:05:47.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.852+0000 7f6d9a7fc640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6d880a2880 0x7f6d880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:47.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.852+0000 7f6d9a7fc640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6d880a2880 0x7f6d880a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c197de0 tx=0x7f6d84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:47.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:47.856+0000 7f6da08bd640 1 -- 172.21.15.78:0/1393795070 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d940d3070 con 0x7f6d9c100700 2024-03-16T11:05:48.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.032+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d9c0623e0 con 0x7f6d9c100700 2024-03-16T11:05:48.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.033+0000 7f6da08bd640 1 -- 172.21.15.78:0/1393795070 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f6d9409be10 con 0x7f6d9c100700 2024-03-16T11:05:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:48.034 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:48.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6d880a2880 msgr2=0x7f6d880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6d880a2880 0x7f6d880a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c197de0 tx=0x7f6d84005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:48.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 msgr2=0x7f6d9c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 0x7f6d9c1968d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400e9c0 tx=0x7f6d9400ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 shutdown_connections 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f6d880a2880 0x7f6d880a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c1022b0 0x7f6d9c19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d9c1018f0 0x7f6d9c196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 --2- 172.21.15.78:0/1393795070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c100700 0x7f6d9c1968d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 >> 172.21.15.78:0/1393795070 conn(0x7f6d9c0fbc90 msgr2=0x7f6d9c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 shutdown_connections 2024-03-16T11:05:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.035+0000 7f6da18bf640 1 -- 172.21.15.78:0/1393795070 wait complete. 2024-03-16T11:05:48.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T11:03:59.617240+0000', 'modified': '2024-03-16T11:04:03.658334+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24505, 'mds_2': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24458': {'gid': 24458, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6835/1190197046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1190197046}, {'type': 'v1', 'addr': '172.21.15.98:6835', 'nonce': 1190197046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/1109965992', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 1109965992}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 1109965992}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/3505861862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 3505861862}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 3505861862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.98:6837/1752557490', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6836', 'nonce': 1752557490}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 1752557490}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.153:6839/1504943793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 1504943793}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 1504943793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6839/4092843053', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 4092843053}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 4092843053}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24488, 24505, 24521]} 2024-03-16T11:05:48.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T11:05:48.048 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T11:05:48.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.150+0000 7f9bec537640 1 -- 172.21.15.78:0/3578270409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be41006e0 msgr2=0x7f9be4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.150+0000 7f9bec537640 1 --2- 172.21.15.78:0/3578270409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be41006e0 0x7f9be4100b00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009ca0 tx=0x7f9be002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.150+0000 7f9bec537640 1 -- 172.21.15.78:0/3578270409 shutdown_connections 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 --2- 172.21.15.78:0/3578270409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9be4102290 0x7f9be4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 --2- 172.21.15.78:0/3578270409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9be41018d0 0x7f9be4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 --2- 172.21.15.78:0/3578270409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be41006e0 0x7f9be4100b00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 -- 172.21.15.78:0/3578270409 >> 172.21.15.78:0/3578270409 conn(0x7f9be40fbc70 msgr2=0x7f9be40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 -- 172.21.15.78:0/3578270409 shutdown_connections 2024-03-16T11:05:48.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 -- 172.21.15.78:0/3578270409 wait complete. 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 Processor -- start 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 -- start start 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9be41006e0 0x7f9be4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9be41018d0 0x7f9be4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be4102290 0x7f9be419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9be410c200 con 0x7f9be4102290 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9be410c080 con 0x7f9be41006e0 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bec537640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9be410c380 con 0x7f9be41018d0 2024-03-16T11:05:48.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9beaad4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be4102290 0x7f9be419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9bea2d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9be41006e0 0x7f9be4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9beaad4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be4102290 0x7f9be419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35986/0 (socket says 172.21.15.78:35986) 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9beaad4640 1 -- 172.21.15.78:0/533769401 learned_addr learned my addr 172.21.15.78:0/533769401 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.151+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9be41018d0 0x7f9be4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9beaad4640 1 -- 172.21.15.78:0/533769401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9be41018d0 msgr2=0x7f9be4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9beaad4640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9be41018d0 0x7f9be4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9beaad4640 1 -- 172.21.15.78:0/533769401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9be41006e0 msgr2=0x7f9be4197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9beaad4640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9be41006e0 0x7f9be4197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9beaad4640 1 -- 172.21.15.78:0/533769401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be419f700 con 0x7f9be4102290 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9be41006e0 0x7f9be4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9beaad4640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be4102290 0x7f9be419f090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00e970 tx=0x7f9bdc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9bdb7fe640 1 -- 172.21.15.78:0/533769401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc00ccc0 con 0x7f9be4102290 2024-03-16T11:05:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be419f9f0 con 0x7f9be4102290 2024-03-16T11:05:48.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9bdb7fe640 1 -- 172.21.15.78:0/533769401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bdc00ce60 con 0x7f9be4102290 2024-03-16T11:05:48.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9bdb7fe640 1 -- 172.21.15.78:0/533769401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc010430 con 0x7f9be4102290 2024-03-16T11:05:48.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.152+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be419ff40 con 0x7f9be4102290 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.154+0000 7f9bdb7fe640 1 -- 172.21.15.78:0/533769401 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9bdc004ff0 con 0x7f9be4102290 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.154+0000 7f9bef1fc740 1 Processor -- start 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bef1fc740 1 -- start start 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f48eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f4940c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f496bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f12f3f2c80 con 0x55f12f494730 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f12f39baa0 con 0x55f12f48f5c0 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f12f3eed20 con 0x55f12f48ebd0 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bdb7fe640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f4940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9be9ad2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f48eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f4940c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:53380/0 (socket says 172.21.15.78:53380) 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 -- 172.21.15.78:0/1391906132 learned_addr learned my addr 172.21.15.78:0/1391906132 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f496bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9beaad4640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bdb7fe640 1 -- 172.21.15.78:0/533769401 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9bdc0cee70 con 0x7f9be4102290 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 -- 172.21.15.78:0/1391906132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 msgr2=0x55f12f48eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f48eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 -- 172.21.15.78:0/1391906132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 msgr2=0x55f12f496bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f496bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 -- 172.21.15.78:0/1391906132 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f12f3bdff0 con 0x55f12f48f5c0 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f496bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9beaad4640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0a3a80 tx=0x7f9bdc0665d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.155+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f4940c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9be002c700 tx=0x7f9be0005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ce0e6774ef31a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bd8bfe640 1 -- 172.21.15.78:0/1391906132 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0043070 con 0x55f12f48f5c0 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bd8bfe640 1 -- 172.21.15.78:0/1391906132 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be0003e20 con 0x55f12f48f5c0 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bd8bfe640 1 -- 172.21.15.78:0/1391906132 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0039040 con 0x55f12f48f5c0 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1391906132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 msgr2=0x55f12f4940c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f4940c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9be002c700 tx=0x7f9be0005ef0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1391906132 shutdown_connections 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f496bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f4940c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1391906132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f48eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.156+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1391906132 >> 172.21.15.78:0/1391906132 conn(0x55f12f48a140 msgr2=0x55f12f48a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1391906132 shutdown_connections 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1391906132 wait complete. 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 -- start start 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 Processor -- start 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f48ebd0 0x55f12f498d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f499260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f494730 0x55f12f4a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f12f497680 con 0x55f12f48ebd0 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f12f3aeef0 con 0x55f12f48f5c0 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f12f497800 con 0x55f12f494730 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bef1fc740 10 client.0 _ll_register_callbacks cb 0x55f12f3a9dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9be9ad2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f48ebd0 0x55f12f498d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9be9ad2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f48ebd0 0x55f12f498d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36000/0 (socket says 172.21.15.78:36000) 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9be9ad2640 1 -- 172.21.15.78:0/1721144836 learned_addr learned my addr 172.21.15.78:0/1721144836 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f499260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f494730 0x55f12f4a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bea2d3640 1 -- 172.21.15.78:0/1721144836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f494730 msgr2=0x55f12f4a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f494730 0x55f12f4a02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bea2d3640 1 -- 172.21.15.78:0/1721144836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f48ebd0 msgr2=0x55f12f498d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f48ebd0 0x55f12f498d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9bea2d3640 1 -- 172.21.15.78:0/1721144836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f12f3bba30 con 0x55f12f48f5c0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.157+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f48ebd0 0x55f12f498d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f499260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55f12f48b0b0 tx=0x7f9be002c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a5d6441e9720b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bcdffb640 1 -- 172.21.15.78:0/1721144836 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be004c030 con 0x55f12f48f5c0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bcdffb640 1 -- 172.21.15.78:0/1721144836 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9be0003b80 con 0x55f12f48f5c0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9be0003e20 con 0x55f12f48f5c0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bcdffb640 1 -- 172.21.15.78:0/1721144836 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9be0039070 con 0x55f12f48f5c0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 10 client.25073 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f12f1072d0 con 0x55f12f48f5c0 2024-03-16T11:05:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bcdffb640 1 -- 172.21.15.78:0/1721144836 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be0003e20 con 0x55f12f48f5c0 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bcdffb640 1 -- 172.21.15.78:0/1721144836 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9be00488e0 con 0x55f12f48f5c0 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 10 client.25073 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 10 client.25073 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 20 client.25073 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f4a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 4 client.25073 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.158+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f12f4001c0 con 0x55f12f4a5270 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.159+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f4a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.159+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f4a66e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eea6eac16664a75e server_cookie=2bf38bcbd4f07d5a in_seq=0 out_seq=0 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.159+0000 7f9bcdffb640 10 client.25073 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bcdffb640 1 -- 172.21.15.78:0/1721144836 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f12f4001c0 con 0x55f12f4a5270 2024-03-16T11:05:48.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bcdffb640 10 client.25073 handle_command_reply: tid=0 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 client.25073 shutdown 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 msgr2=0x55f12f499260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f499260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55f12f48b0b0 tx=0x7f9be002c870 comp rx=0 tx=0).stop 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 shutdown_connections 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f4a66e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f494730 0x55f12f4a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f499260 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bcdffb640 0 client.25073 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1721144836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f48ebd0 0x55f12f498d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 >> 172.21.15.78:0/1721144836 conn(0x55f12f48a140 msgr2=0x55f12f489380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 shutdown_connections 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1721144836 wait complete. 2024-03-16T11:05:48.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.160+0000 7f9bef1fc740 20 client.25073 trim_cache size 0 max 16384 2024-03-16T11:05:48.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.177+0000 7f9bef1fc740 1 Processor -- start 2024-03-16T11:05:48.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.177+0000 7f9bef1fc740 1 -- start start 2024-03-16T11:05:48.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48ebd0 0x55f12f496b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48f5c0 0x55f12f4a0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f4991c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f12f4a4a20 con 0x55f12f494730 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f12f4a50f0 con 0x55f12f48ebd0 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f12f4a57c0 con 0x55f12f48f5c0 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f4991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f4991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36002/0 (socket says 172.21.15.78:36002) 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 -- 172.21.15.78:0/1705197378 learned_addr learned my addr 172.21.15.78:0/1705197378 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48f5c0 0x55f12f4a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48ebd0 0x55f12f496b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 -- 172.21.15.78:0/1705197378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48f5c0 msgr2=0x55f12f4a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48f5c0 0x55f12f4a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 -- 172.21.15.78:0/1705197378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48ebd0 msgr2=0x55f12f496b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48ebd0 0x55f12f496b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 -- 172.21.15.78:0/1705197378 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f12f1d16b0 con 0x55f12f494730 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48f5c0 0x55f12f4a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f4991c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4026340 tx=0x7f9bd4013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1cc89cc8fa744c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bcffff640 1 -- 172.21.15.78:0/1705197378 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4004050 con 0x55f12f494730 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bcffff640 1 -- 172.21.15.78:0/1705197378 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd400fc70 con 0x55f12f494730 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.178+0000 7f9bcffff640 1 -- 172.21.15.78:0/1705197378 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4003a10 con 0x55f12f494730 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1705197378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 msgr2=0x55f12f4991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f4991c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4026340 tx=0x7f9bd4013040 comp rx=0 tx=0).stop 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1705197378 shutdown_connections 2024-03-16T11:05:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f4991c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48f5c0 0x55f12f4a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/1705197378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48ebd0 0x55f12f496b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1705197378 >> 172.21.15.78:0/1705197378 conn(0x55f12f48a140 msgr2=0x55f12f48a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1705197378 shutdown_connections 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- 172.21.15.78:0/1705197378 wait complete. 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- start start 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 Processor -- start 2024-03-16T11:05:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f49b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f49b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f49e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f12f4a6560 con 0x55f12f494730 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f12f4a5e90 con 0x55f12f48f5c0 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.179+0000 7f9bef1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f12f4a6c30 con 0x55f12f48ebd0 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bef1fc740 10 client.0 _ll_register_callbacks cb 0x7f9bd40099f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f49e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f49e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36006/0 (socket says 172.21.15.78:36006) 2024-03-16T11:05:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 -- 172.21.15.78:0/3551428237 learned_addr learned my addr 172.21.15.78:0/3551428237 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f49b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f49b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 -- 172.21.15.78:0/3551428237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 msgr2=0x55f12f49b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f49b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 -- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 msgr2=0x55f12f49b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f49b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 -- 172.21.15.78:0/3551428237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f12f49eef0 con 0x55f12f494730 2024-03-16T11:05:48.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9be9ad2640 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f49e900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40058c0 tx=0x7f9bd4009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586c70230cbbb46a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:48.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bcdffb640 1 -- 172.21.15.78:0/3551428237 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd400fe20 con 0x55f12f494730 2024-03-16T11:05:48.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bcdffb640 1 -- 172.21.15.78:0/3551428237 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9bd400f7c0 con 0x55f12f494730 2024-03-16T11:05:48.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bd400fc70 con 0x55f12f494730 2024-03-16T11:05:48.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bcdffb640 1 -- 172.21.15.78:0/3551428237 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9bd400f9c0 con 0x55f12f494730 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bef1fc740 10 client.15243 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f12f4a01f0 con 0x55f12f494730 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.180+0000 7f9bcdffb640 1 -- 172.21.15.78:0/3551428237 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd400fc70 con 0x55f12f494730 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bcdffb640 1 -- 172.21.15.78:0/3551428237 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9bd403e050 con 0x55f12f494730 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 10 client.15243 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 10 client.15243 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 20 client.15243 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:48.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f50bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 4 client.15243 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:05:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "status"}) -- 0x55f12f3fa510 con 0x55f12f4a5270 2024-03-16T11:05:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f50bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bcdffb640 10 client.15243 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bea2d3640 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f50bd70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed44713e1bbf3b6f server_cookie=141ff72a2cb8466f in_seq=0 out_seq=0 2024-03-16T11:05:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bcdffb640 1 -- 172.21.15.78:0/3551428237 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55f12f3fa510 con 0x55f12f4a5270 2024-03-16T11:05:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bcdffb640 10 client.15243 handle_command_reply: tid=0 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 client.15243 shutdown 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 msgr2=0x55f12f49e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f49e900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40058c0 tx=0x7f9bd4009aa0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 shutdown_connections 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55f12f4a5270 0x55f12f50bd70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f12f494730 0x55f12f49e900 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bcdffb640 0 client.15243 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:05:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f12f48f5c0 0x55f12f49b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 --2- 172.21.15.78:0/3551428237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f12f48ebd0 0x55f12f49b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.181+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 >> 172.21.15.78:0/3551428237 conn(0x55f12f48a140 msgr2=0x55f12f48d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 shutdown_connections 2024-03-16T11:05:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bef1fc740 1 -- 172.21.15.78:0/3551428237 wait complete. 2024-03-16T11:05:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bef1fc740 20 client.15243 trim_cache size 0 max 16384 2024-03-16T11:05:48.195 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:05:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T11:05:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-03-16T11:05:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24488, 2024-03-16T11:05:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-16T11:05:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-16T11:05:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-16T11:05:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 104.877497507, 2024-03-16T11:05:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 21, 2024-03-16T11:05:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 90, 2024-03-16T11:05:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 88, 2024-03-16T11:05:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 124.306506083 2024-03-16T11:05:48.197 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:05:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9bd00a2800 msgr2=0x7f9bd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0a3a80 tx=0x7f9bdc0665d0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be4102290 msgr2=0x7f9be419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be4102290 0x7f9be419f090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00e970 tx=0x7f9bdc00ed70 comp rx=0 tx=0).stop 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 shutdown_connections 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9be4102290 0x7f9be419f090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9bd00a2800 0x7f9bd00a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9be41018d0 0x7f9be4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 --2- 172.21.15.78:0/533769401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9be41006e0 0x7f9be4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 >> 172.21.15.78:0/533769401 conn(0x7f9be40fbc70 msgr2=0x7f9be40689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 shutdown_connections 2024-03-16T11:05:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.182+0000 7f9bec537640 1 -- 172.21.15.78:0/533769401 wait complete. 2024-03-16T11:05:48.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T11:05:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 -- 172.21.15.78:0/1844248939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 msgr2=0x7f3b58101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/1844248939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b58101d60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009900 tx=0x7f3b4c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 -- 172.21.15.78:0/1844248939 shutdown_connections 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/1844248939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581022a0 0x7f3b58108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/1844248939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b58101d60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/1844248939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b581006f0 0x7f3b58100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 -- 172.21.15.78:0/1844248939 >> 172.21.15.78:0/1844248939 conn(0x7f3b580fbc80 msgr2=0x7f3b580fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 -- 172.21.15.78:0/1844248939 shutdown_connections 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 -- 172.21.15.78:0/1844248939 wait complete. 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.303+0000 7f3b5db1b640 1 Processor -- start 2024-03-16T11:05:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 -- start start 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581006f0 0x7f3b5806a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b5819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b581022a0 0x7f3b5819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5810c220 con 0x7f3b581018e0 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5810c0a0 con 0x7f3b581022a0 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5810c3a0 con 0x7f3b581006f0 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b5819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581006f0 0x7f3b5806a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b5819c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36012/0 (socket says 172.21.15.78:36012) 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 -- 172.21.15.78:0/2635630844 learned_addr learned my addr 172.21.15.78:0/2635630844 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b57fff640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b581022a0 0x7f3b5819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 -- 172.21.15.78:0/2635630844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581006f0 msgr2=0x7f3b5806a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581006f0 0x7f3b5806a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 -- 172.21.15.78:0/2635630844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b581022a0 msgr2=0x7f3b5819edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b581022a0 0x7f3b5819edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 -- 172.21.15.78:0/2635630844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5819f6d0 con 0x7f3b581018e0 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b577fe640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581006f0 0x7f3b5806a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b5819c7a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c006010 tx=0x7f3b4c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b54ff9640 1 -- 172.21.15.78:0/2635630844 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c043020 con 0x7f3b581018e0 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5819f900 con 0x7f3b581018e0 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b54ff9640 1 -- 172.21.15.78:0/2635630844 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b4c0043d0 con 0x7f3b581018e0 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b54ff9640 1 -- 172.21.15.78:0/2635630844 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c003b80 con 0x7f3b581018e0 2024-03-16T11:05:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.304+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b5819fe50 con 0x7f3b581018e0 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.306+0000 7f3b54ff9640 1 -- 172.21.15.78:0/2635630844 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3b4c046020 con 0x7f3b581018e0 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 Processor -- start 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b54ff9640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b400a2800 0x7f3b400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 -- start start 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b57fff640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b400a2800 0x7f3b400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b54ff9640 1 -- 172.21.15.78:0/2635630844 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f3b4c0f0cb0 con 0x7f3b581018e0 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b02d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b02d5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5605b0231c80 con 0x5605b02d3730 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5605b01daaa0 con 0x5605b02cdbd0 2024-03-16T11:05:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b607e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5605b022dd20 con 0x5605b02ce5c0 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b577fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b57fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b02d5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b57fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b02d5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36016/0 (socket says 172.21.15.78:36016) 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b577fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:53430/0 (socket says 172.21.15.78:53430) 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b57fff640 1 -- 172.21.15.78:0/1555808597 learned_addr learned my addr 172.21.15.78:0/1555808597 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b02d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.307+0000 7f3b57fff640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b400a2800 0x7f3b400a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3b50005fd0 tx=0x7f3b50011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b577fe640 1 -- 172.21.15.78:0/1555808597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 msgr2=0x5605b02d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b577fe640 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b02d30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b577fe640 1 -- 172.21.15.78:0/1555808597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 msgr2=0x5605b02d5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b577fe640 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b02d5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b577fe640 1 -- 172.21.15.78:0/1555808597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b01fcff0 con 0x5605b02cdbd0 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b57fff640 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b02d5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b577fe640 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02cdff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009fa0 tx=0x7f3b480118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440fa0fcabe70ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b3dffb640 1 -- 172.21.15.78:0/1555808597 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48004310 con 0x5605b02cdbd0 2024-03-16T11:05:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b3dffb640 1 -- 172.21.15.78:0/1555808597 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b480044b0 con 0x5605b02cdbd0 2024-03-16T11:05:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.308+0000 7f3b3dffb640 1 -- 172.21.15.78:0/1555808597 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480035f0 con 0x5605b02cdbd0 2024-03-16T11:05:48.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:48 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1765278114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:48.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:48 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2309683328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:48.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:48 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3654281853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:48.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:48 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1393795070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:48.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 -- 172.21.15.78:0/1555808597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 msgr2=0x5605b02cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02cdff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009fa0 tx=0x7f3b480118d0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 -- 172.21.15.78:0/1555808597 shutdown_connections 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b02d5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b02d30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 --2- 172.21.15.78:0/1555808597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02cdff0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 -- 172.21.15.78:0/1555808597 >> 172.21.15.78:0/1555808597 conn(0x5605b02c9140 msgr2=0x5605b02c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 -- 172.21.15.78:0/1555808597 shutdown_connections 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 -- 172.21.15.78:0/1555808597 wait complete. 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 -- start start 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.309+0000 7f3b607e9740 1 Processor -- start 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02d8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02df2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5605b02d66e0 con 0x5605b02ce5c0 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5605b01edef0 con 0x5605b02cdbd0 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5605b02d6860 con 0x5605b02d3730 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 10 client.0 _ll_register_callbacks cb 0x5605b01e8dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02d8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02d8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36022/0 (socket says 172.21.15.78:36022) 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 -- 172.21.15.78:0/3551797045 learned_addr learned my addr 172.21.15.78:0/3551797045 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b577fe640 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b57fff640 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02df2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 -- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 msgr2=0x5605b02df2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02df2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 -- 172.21.15.78:0/3551797045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 msgr2=0x5605b02d7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 -- 172.21.15.78:0/3551797045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b01faa30 con 0x5605b02ce5c0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02d8320 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c005ab0 tx=0x7f3b4c0bd030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c530a85e663c1fed server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b36ffd640 1 -- 172.21.15.78:0/3551797045 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0c1760 con 0x5605b02ce5c0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b36ffd640 1 -- 172.21.15.78:0/3551797045 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f3b4c0054b0 con 0x5605b02ce5c0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b480044b0 con 0x5605b02ce5c0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b36ffd640 1 -- 172.21.15.78:0/3551797045 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3b4c0056b0 con 0x5605b02ce5c0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 10 client.15255 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.310+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5605aff462d0 con 0x5605b02ce5c0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b36ffd640 1 -- 172.21.15.78:0/3551797045 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b480044b0 con 0x5605b02ce5c0 2024-03-16T11:05:48.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b36ffd640 1 -- 172.21.15.78:0/3551797045 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3b4c0c5920 con 0x5605b02ce5c0 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b607e9740 10 client.15255 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b607e9740 10 client.15255 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b607e9740 20 client.15255 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b607e9740 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b02e4880 0x5605b02e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b607e9740 4 client.15255 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605b023f1c0 con 0x5605b02e4880 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b577fe640 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b02e4880 0x5605b02e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b577fe640 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b02e4880 0x5605b02e5cf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=262c901b56a02342 server_cookie=86389c6c446ad070 in_seq=0 out_seq=0 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.311+0000 7f3b36ffd640 10 client.15255 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b36ffd640 1 -- 172.21.15.78:0/3551797045 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605b023f1c0 con 0x5605b02e4880 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b36ffd640 10 client.15255 handle_command_reply: tid=0 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 client.15255 shutdown 2024-03-16T11:05:48.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 msgr2=0x5605b02d8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02d8320 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c005ab0 tx=0x7f3b4c0bd030 comp rx=0 tx=0).stop 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 shutdown_connections 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b02e4880 0x5605b02e5cf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02df2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02d8320 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b36ffd640 0 client.15255 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 --2- 172.21.15.78:0/3551797045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 >> 172.21.15.78:0/3551797045 conn(0x5605b02c9140 msgr2=0x5605b02c8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 shutdown_connections 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 1 -- 172.21.15.78:0/3551797045 wait complete. 2024-03-16T11:05:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.312+0000 7f3b607e9740 20 client.15255 trim_cache size 0 max 16384 2024-03-16T11:05:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 Processor -- start 2024-03-16T11:05:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 -- start start 2024-03-16T11:05:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02c8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5605b02e4030 con 0x5605b02ce5c0 2024-03-16T11:05:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5605b02e4700 con 0x5605b02cdbd0 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b607e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5605b02e4dd0 con 0x5605b02d3730 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36034/0 (socket says 172.21.15.78:36034) 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 -- 172.21.15.78:0/626584121 learned_addr learned my addr 172.21.15.78:0/626584121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b577fe640 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b57fff640 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 -- 172.21.15.78:0/626584121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 msgr2=0x5605b02c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 -- 172.21.15.78:0/626584121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 msgr2=0x5605b02d8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b56ffd640 1 -- 172.21.15.78:0/626584121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b00106b0 con 0x5605b02ce5c0 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.330+0000 7f3b57fff640 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02c7af0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c005c20 tx=0x7f3b4c0b99e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef001b4523fff83c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b3d7fa640 1 -- 172.21.15.78:0/626584121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0c1760 con 0x5605b02ce5c0 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b3d7fa640 1 -- 172.21.15.78:0/626584121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b4c0d6070 con 0x5605b02ce5c0 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b3d7fa640 1 -- 172.21.15.78:0/626584121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0bd870 con 0x5605b02ce5c0 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 -- 172.21.15.78:0/626584121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 msgr2=0x5605b02c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02c7af0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c005c20 tx=0x7f3b4c0b99e0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 -- 172.21.15.78:0/626584121 shutdown_connections 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02d3730 0x5605b02c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02ce5c0 0x5605b02c7af0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 --2- 172.21.15.78:0/626584121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b02d8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 -- 172.21.15.78:0/626584121 >> 172.21.15.78:0/626584121 conn(0x5605b02c9140 msgr2=0x5605b02d2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 -- 172.21.15.78:0/626584121 shutdown_connections 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.331+0000 7f3b607e9740 1 -- 172.21.15.78:0/626584121 wait complete. 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 -- start start 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 Processor -- start 2024-03-16T11:05:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b03485e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b034ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b034d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5605b02e5b70 con 0x5605b02d3730 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5605b02e54a0 con 0x5605b02cdbd0 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5605b02e6240 con 0x5605b02ce5c0 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b607e9740 10 client.0 _ll_register_callbacks cb 0x7f3b4c0b9930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b034d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b034d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36048/0 (socket says 172.21.15.78:36048) 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 -- 172.21.15.78:0/2832183347 learned_addr learned my addr 172.21.15.78:0/2832183347 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b56ffd640 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b034ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b577fe640 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b03485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 -- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 msgr2=0x5605b034ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b034ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 -- 172.21.15.78:0/2832183347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 msgr2=0x5605b03485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b03485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 -- 172.21.15.78:0/2832183347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b034d610 con 0x5605b02d3730 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b57fff640 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b034d080 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3b5001dc90 tx=0x7f3b50016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add0db9a9ec249a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.332+0000 7f3b36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b36ffd640 1 -- 172.21.15.78:0/2832183347 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50017040 con 0x5605b02d3730 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b36ffd640 1 -- 172.21.15.78:0/2832183347 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f3b5000fcf0 con 0x5605b02d3730 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b4c0d6070 con 0x5605b02d3730 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b36ffd640 1 -- 172.21.15.78:0/2832183347 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3b50003970 con 0x5605b02d3730 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 10 client.15261 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5605b034e9e0 con 0x5605b02d3730 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b36ffd640 1 -- 172.21.15.78:0/2832183347 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b4c0d6070 con 0x5605b02d3730 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b36ffd640 1 -- 172.21.15.78:0/2832183347 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3b500323f0 con 0x5605b02d3730 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 10 client.15261 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 10 client.15261 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 20 client.15261 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b03536b0 0x5605b034faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 4 client.15261 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "status"}) -- 0x5605b0239510 con 0x5605b03536b0 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.333+0000 7f3b577fe640 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b03536b0 0x5605b034faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b36ffd640 10 client.15261 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b577fe640 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b03536b0 0x5605b034faf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b8a152d36c06f636 server_cookie=376cda5239300f80 in_seq=0 out_seq=0 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b36ffd640 1 -- 172.21.15.78:0/2832183347 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5605b0239510 con 0x5605b03536b0 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b36ffd640 10 client.15261 handle_command_reply: tid=0 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 client.15261 shutdown 2024-03-16T11:05:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 msgr2=0x5605b034d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b034d080 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3b5001dc90 tx=0x7f3b50016000 comp rx=0 tx=0).stop 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 shutdown_connections 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5605b03536b0 0x5605b034faf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605b02d3730 0x5605b034d080 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605b02ce5c0 0x5605b034ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b36ffd640 0 client.15261 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 --2- 172.21.15.78:0/2832183347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5605b02cdbd0 0x5605b03485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 >> 172.21.15.78:0/2832183347 conn(0x5605b02c9140 msgr2=0x5605b02cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 shutdown_connections 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 1 -- 172.21.15.78:0/2832183347 wait complete. 2024-03-16T11:05:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.334+0000 7f3b607e9740 20 client.15261 trim_cache size 0 max 16384 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24505, 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 106.035002514, 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 21, 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 90, 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 88, 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 121.813266082 2024-03-16T11:05:48.343 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:05:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b400a2800 msgr2=0x7f3b400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b400a2800 0x7f3b400a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3b50005fd0 tx=0x7f3b50011810 comp rx=0 tx=0).stop 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 msgr2=0x7f3b5819c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b5819c7a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c006010 tx=0x7f3b4c005e40 comp rx=0 tx=0).stop 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 shutdown_connections 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b581022a0 0x7f3b5819edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b581018e0 0x7f3b5819c7a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f3b400a2800 0x7f3b400a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 --2- 172.21.15.78:0/2635630844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581006f0 0x7f3b5806a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 >> 172.21.15.78:0/2635630844 conn(0x7f3b580fbc80 msgr2=0x7f3b58104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 shutdown_connections 2024-03-16T11:05:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.335+0000 7f3b5db1b640 1 -- 172.21.15.78:0/2635630844 wait complete. 2024-03-16T11:05:48.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T11:05:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.448+0000 7f8ae3a75640 1 -- 172.21.15.78:0/1894424914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1018d0 msgr2=0x7f8adc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.448+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/1894424914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1018d0 0x7f8adc101d50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009900 tx=0x7f8acc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- 172.21.15.78:0/1894424914 shutdown_connections 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/1894424914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8adc102290 0x7f8adc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/1894424914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1018d0 0x7f8adc101d50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/1894424914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8adc1006e0 0x7f8adc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- 172.21.15.78:0/1894424914 >> 172.21.15.78:0/1894424914 conn(0x7f8adc0fbc70 msgr2=0x7f8adc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- 172.21.15.78:0/1894424914 shutdown_connections 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- 172.21.15.78:0/1894424914 wait complete. 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 Processor -- start 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- start start 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1006e0 0x7f8adc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8adc1018d0 0x7f8adc19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8adc102290 0x7f8adc19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc10c210 con 0x7f8adc1006e0 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc10c090 con 0x7f8adc102290 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae3a75640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc10c390 con 0x7f8adc1018d0 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae1811640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1006e0 0x7f8adc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae1811640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1006e0 0x7f8adc0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36068/0 (socket says 172.21.15.78:36068) 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae1811640 1 -- 172.21.15.78:0/2160491972 learned_addr learned my addr 172.21.15.78:0/2160491972 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8adc102290 0x7f8adc19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.449+0000 7f8ae1010640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8adc1018d0 0x7f8adc19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae1811640 1 -- 172.21.15.78:0/2160491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8adc1018d0 msgr2=0x7f8adc19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8adc1018d0 0x7f8adc19c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae1811640 1 -- 172.21.15.78:0/2160491972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8adc102290 msgr2=0x7f8adc19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8adc102290 0x7f8adc19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae1811640 1 -- 172.21.15.78:0/2160491972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8adc19f6c0 con 0x7f8adc1006e0 2024-03-16T11:05:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1006e0 0x7f8adc0694b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800e9c0 tx=0x7f8ad800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ad2ffd640 1 -- 172.21.15.78:0/2160491972 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800cd80 con 0x7f8adc1006e0 2024-03-16T11:05:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ad2ffd640 1 -- 172.21.15.78:0/2160491972 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ad80035f0 con 0x7f8adc1006e0 2024-03-16T11:05:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ad2ffd640 1 -- 172.21.15.78:0/2160491972 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8010690 con 0x7f8adc1006e0 2024-03-16T11:05:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8adc19f950 con 0x7f8adc1006e0 2024-03-16T11:05:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.450+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8adc19fe70 con 0x7f8adc1006e0 2024-03-16T11:05:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.452+0000 7f8ad2ffd640 1 -- 172.21.15.78:0/2160491972 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8ad80037a0 con 0x7f8adc1006e0 2024-03-16T11:05:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 Processor -- start 2024-03-16T11:05:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 -- start start 2024-03-16T11:05:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ad2ffd640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8ac80a2800 0x7f8ac80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f83ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f845c0 0x55b531f890c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f89730 0x55b531f8bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b531ee7c80 con 0x55b531f83bd0 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b531e90aa0 con 0x55b531f89730 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae6659740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b531ee3d20 con 0x55b531f845c0 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8ac80a2800 0x7f8ac80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1811640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f83ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1010640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f845c0 0x55b531f890c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1811640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f83ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36084/0 (socket says 172.21.15.78:36084) 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1811640 1 -- 172.21.15.78:0/1822660346 learned_addr learned my addr 172.21.15.78:0/1822660346 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae2012640 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f89730 0x55b531f8bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ad2ffd640 1 -- 172.21.15.78:0/2160491972 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8ad80cedf0 con 0x7f8adc1006e0 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1010640 1 -- 172.21.15.78:0/1822660346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f89730 msgr2=0x55b531f8bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1010640 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f89730 0x55b531f8bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1010640 1 -- 172.21.15.78:0/1822660346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 msgr2=0x55b531f83ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1010640 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f83ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1010640 1 -- 172.21.15.78:0/1822660346 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b531eb2ff0 con 0x55b531f845c0 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.453+0000 7f8ae1811640 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f83ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.454+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8ac80a2800 0x7f8ac80a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4005fd0 tx=0x7f8ad4009e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.454+0000 7f8ae1010640 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f845c0 0x55b531f890c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8acc000c00 tx=0x7f8acc009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a3954662ac39827 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.454+0000 7f8ac7fff640 1 -- 172.21.15.78:0/1822660346 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc045020 con 0x55b531f845c0 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.454+0000 7f8ac7fff640 1 -- 172.21.15.78:0/1822660346 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8acc004040 con 0x55b531f845c0 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.454+0000 7f8ac7fff640 1 -- 172.21.15.78:0/1822660346 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc005560 con 0x55b531f845c0 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- 172.21.15.78:0/1822660346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f845c0 msgr2=0x55b531f890c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f845c0 0x55b531f890c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8acc000c00 tx=0x7f8acc009560 comp rx=0 tx=0).stop 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- 172.21.15.78:0/1822660346 shutdown_connections 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f89730 0x55b531f8bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f845c0 0x55b531f890c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 --2- 172.21.15.78:0/1822660346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f83ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- 172.21.15.78:0/1822660346 >> 172.21.15.78:0/1822660346 conn(0x55b531f7f140 msgr2=0x55b531f7f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- 172.21.15.78:0/1822660346 shutdown_connections 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- 172.21.15.78:0/1822660346 wait complete. 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- start start 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 Processor -- start 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f8dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f8e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f95280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b531f8c680 con 0x55b531f89730 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b531ea3ef0 con 0x55b531f845c0 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.455+0000 7f8ae6659740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b531f8c800 con 0x55b531f83bd0 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae6659740 10 client.0 _ll_register_callbacks cb 0x55b531e9edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f95280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f95280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36090/0 (socket says 172.21.15.78:36090) 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 -- 172.21.15.78:0/934987592 learned_addr learned my addr 172.21.15.78:0/934987592 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae1811640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f8dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae1010640 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f8e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 -- 172.21.15.78:0/934987592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 msgr2=0x55b531f8dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f8dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 -- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 msgr2=0x55b531f8e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f8e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 -- 172.21.15.78:0/934987592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b531eb0a30 con 0x55b531f89730 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae2012640 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f95280 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400ffd0 tx=0x7f8ad401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1606bb7bcfe2fe17 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ac57fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ac57fa640 1 -- 172.21.15.78:0/934987592 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4004b20 con 0x55b531f89730 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ac57fa640 1 -- 172.21.15.78:0/934987592 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8ad401c040 con 0x55b531f89730 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8acc004040 con 0x55b531f89730 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ac57fa640 1 -- 172.21.15.78:0/934987592 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8ad4031050 con 0x55b531f89730 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae6659740 10 client.15273 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b531bfc2d0 con 0x55b531f89730 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.456+0000 7f8ac57fa640 1 -- 172.21.15.78:0/934987592 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8acc004040 con 0x55b531f89730 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ac57fa640 1 -- 172.21.15.78:0/934987592 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8ad4028070 con 0x55b531f89730 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae6659740 10 client.15273 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae6659740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae6659740 20 client.15273 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae6659740 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b531f9a7f0 0x55b531f9bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae6659740 4 client.15273 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:05:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b531ef51c0 con 0x55b531f9a7f0 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae1811640 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b531f9a7f0 0x55b531f9bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ac57fa640 10 client.15273 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.457+0000 7f8ae1811640 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b531f9a7f0 0x55b531f9bc60 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92f084e50d89fb76 server_cookie=63ded4f665651fcb in_seq=0 out_seq=0 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ac57fa640 1 -- 172.21.15.78:0/934987592 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b531ef51c0 con 0x55b531f9a7f0 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ac57fa640 10 client.15273 handle_command_reply: tid=0 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 client.15273 shutdown 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 msgr2=0x55b531f95280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f95280 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400ffd0 tx=0x7f8ad401d000 comp rx=0 tx=0).stop 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 shutdown_connections 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b531f9a7f0 0x55b531f9bc60 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ac57fa640 0 client.15273 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f95280 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f8e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 --2- 172.21.15.78:0/934987592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f8dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 >> 172.21.15.78:0/934987592 conn(0x55b531f7f140 msgr2=0x55b531f7e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 shutdown_connections 2024-03-16T11:05:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 1 -- 172.21.15.78:0/934987592 wait complete. 2024-03-16T11:05:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.458+0000 7f8ae6659740 20 client.15273 trim_cache size 0 max 16384 2024-03-16T11:05:48.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:48 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1765278114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:48.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:48 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2309683328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:48.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:48 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3654281853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:48.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:48 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1393795070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 Processor -- start 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 -- start start 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f7a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f951a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f89730 0x55b531f85060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b531f99fa0 con 0x55b531f83bd0 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b531f9a670 con 0x55b531f845c0 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae6659740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b531f9ad40 con 0x55b531f89730 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f7a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f7a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36100/0 (socket says 172.21.15.78:36100) 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 -- 172.21.15.78:0/2411718126 learned_addr learned my addr 172.21.15.78:0/2411718126 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1010640 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f89730 0x55b531f85060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 -- 172.21.15.78:0/2411718126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f89730 msgr2=0x55b531f85060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f89730 0x55b531f85060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 -- 172.21.15.78:0/2411718126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 msgr2=0x55b531f951a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f951a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 -- 172.21.15.78:0/2411718126 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b531cc66b0 con 0x55b531f83bd0 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f89730 0x55b531f85060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f7a210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8003dd0 tx=0x7f8ad8003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b54e224b654d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ac77fe640 1 -- 172.21.15.78:0/2411718126 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8066c40 con 0x55b531f83bd0 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ac77fe640 1 -- 172.21.15.78:0/2411718126 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ad809b790 con 0x55b531f83bd0 2024-03-16T11:05:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.476+0000 7f8ac77fe640 1 -- 172.21.15.78:0/2411718126 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8097e00 con 0x55b531f83bd0 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 -- 172.21.15.78:0/2411718126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 msgr2=0x55b531f7a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f7a210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8003dd0 tx=0x7f8ad8003e30 comp rx=0 tx=0).stop 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 -- 172.21.15.78:0/2411718126 shutdown_connections 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f89730 0x55b531f85060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f951a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2411718126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f83bd0 0x55b531f7a210 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 -- 172.21.15.78:0/2411718126 >> 172.21.15.78:0/2411718126 conn(0x55b531f7f140 msgr2=0x55b531f8dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 -- 172.21.15.78:0/2411718126 shutdown_connections 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 -- 172.21.15.78:0/2411718126 wait complete. 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 -- start start 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.477+0000 7f8ae6659740 1 Processor -- start 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f90b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f91040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f94060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b531f9bae0 con 0x55b531f89730 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b531f9b410 con 0x55b531f845c0 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b531f9c1b0 con 0x55b531f83bd0 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 10 client.0 _ll_register_callbacks cb 0x7f8ad4009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f94060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f94060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36112/0 (socket says 172.21.15.78:36112) 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 -- 172.21.15.78:0/2804080881 learned_addr learned my addr 172.21.15.78:0/2804080881 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae1010640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f91040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f90b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 -- 172.21.15.78:0/2804080881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 msgr2=0x55b531f90b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f90b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 -- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 msgr2=0x55b531f91040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f91040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 -- 172.21.15.78:0/2804080881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b531f94650 con 0x55b531f89730 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae1010640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f91040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae2012640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f94060 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40106f0 tx=0x7f8ad4009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a8253c6ba97667 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ac57fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ac57fa640 1 -- 172.21.15.78:0/2804080881 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4010430 con 0x55b531f89730 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ac57fa640 1 -- 172.21.15.78:0/2804080881 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8ad4009040 con 0x55b531f89730 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ad809b790 con 0x55b531f89730 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.478+0000 7f8ac57fa640 1 -- 172.21.15.78:0/2804080881 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8ad4013970 con 0x55b531f89730 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 10 client.15279 fetch_fsmap learned FSMap version 21 2024-03-16T11:05:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b532000650 con 0x55b531f89730 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ac57fa640 1 -- 172.21.15.78:0/2804080881 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ad809b790 con 0x55b531f89730 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ac57fa640 1 -- 172.21.15.78:0/2804080881 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8ad4028040 con 0x55b531f89730 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 10 client.15279 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 10 client.15279 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 20 client.15279 populate_metadata read hostname 'smithi078' 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b5320052b0 0x55b532003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 4 client.15279 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "status"}) -- 0x55b531eef510 con 0x55b5320052b0 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b5320052b0 0x55b532003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ac57fa640 10 client.15279 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.479+0000 7f8ae1811640 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b5320052b0 0x55b532003fc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6f341f77d16149b server_cookie=98136bcbe5c8f481 in_seq=0 out_seq=0 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ac57fa640 1 -- 172.21.15.78:0/2804080881 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55b531eef510 con 0x55b5320052b0 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ac57fa640 10 client.15279 handle_command_reply: tid=0 2024-03-16T11:05:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 client.15279 shutdown 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 msgr2=0x55b531f94060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f94060 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40106f0 tx=0x7f8ad4009fb0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 shutdown_connections 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b5320052b0 0x55b532003fc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b531f89730 0x55b531f94060 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b531f845c0 0x55b531f91040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 --2- 172.21.15.78:0/2804080881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b531f83bd0 0x55b531f90b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ac57fa640 0 client.15279 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 >> 172.21.15.78:0/2804080881 conn(0x55b531f7f140 msgr2=0x55b531f823a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 shutdown_connections 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 1 -- 172.21.15.78:0/2804080881 wait complete. 2024-03-16T11:05:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae6659740 20 client.15279 trim_cache size 0 max 16384 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "a1941e5e-e383-11ee-95c9-87774f69a715", 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24521, 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 108.197245623, 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 21, 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 90, 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 88, 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 113.524707298 2024-03-16T11:05:48.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:05:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8ac80a2800 msgr2=0x7f8ac80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8ac80a2800 0x7f8ac80a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4005fd0 tx=0x7f8ad4009e00 comp rx=0 tx=0).stop 2024-03-16T11:05:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1006e0 msgr2=0x7f8adc0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1006e0 0x7f8adc0694b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800e9c0 tx=0x7f8ad800ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 shutdown_connections 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8adc102290 0x7f8adc19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8ac80a2800 0x7f8ac80a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8adc1018d0 0x7f8adc19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 --2- 172.21.15.78:0/2160491972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc1006e0 0x7f8adc0694b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 >> 172.21.15.78:0/2160491972 conn(0x7f8adc0fbc70 msgr2=0x7f8adc104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 shutdown_connections 2024-03-16T11:05:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:48.480+0000 7f8ae3a75640 1 -- 172.21.15.78:0/2160491972 wait complete. 2024-03-16T11:05:48.492 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-16T11:05:48.518 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-16T11:05:48.518 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-16T11:05:48.518 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-16T11:05:48.518 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T11:05:48.518 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T11:05:48.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:48.518 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-16T11:05:48.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:48.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-16T11:05:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:48 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1765278114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:48 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2309683328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:48 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3654281853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:48.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:48 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1393795070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:Cannot find device "ceph-brx" 2024-03-16T11:05:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:05:48.604 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-16T11:05:48.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T11:05:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:05:49.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:49 smithi078 ceph-mon[32235]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T11:05:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:49 smithi098 ceph-mon[35848]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T11:05:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:49 smithi153 ceph-mon[35521]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 -- 172.21.15.78:0/1688897758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 msgr2=0x7fd07410d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 --2- 172.21.15.78:0/1688897758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd07410d810 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00b0a0 tx=0x7fd06c00b3b0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 -- 172.21.15.78:0/1688897758 shutdown_connections 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 --2- 172.21.15.78:0/1688897758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd07410d810 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 --2- 172.21.15.78:0/1688897758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0741069f0 0x7fd074106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 --2- 172.21.15.78:0/1688897758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd074100c10 0x7fd074101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 -- 172.21.15.78:0/1688897758 >> 172.21.15.78:0/1688897758 conn(0x7fd074076530 msgr2=0x7fd074076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 -- 172.21.15.78:0/1688897758 shutdown_connections 2024-03-16T11:05:50.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.612+0000 7fd07a283700 1 -- 172.21.15.78:0/1688897758 wait complete. 2024-03-16T11:05:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 Processor -- start 2024-03-16T11:05:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 -- start start 2024-03-16T11:05:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd074100c10 0x7fd0741966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd074196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0741069f0 0x7fd07419af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd074110290 con 0x7fd0741015d0 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd074110110 con 0x7fd074100c10 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd07a283700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd074110410 con 0x7fd0741069f0 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd0737fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd074196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd0737fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd074196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36124/0 (socket says 172.21.15.78:36124) 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd0737fe700 1 -- 172.21.15.78:0/641715370 learned_addr learned my addr 172.21.15.78:0/641715370 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd073fff700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd074100c10 0x7fd0741966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.613+0000 7fd078820700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0741069f0 0x7fd07419af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0737fe700 1 -- 172.21.15.78:0/641715370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0741069f0 msgr2=0x7fd07419af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0737fe700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0741069f0 0x7fd07419af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0737fe700 1 -- 172.21.15.78:0/641715370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd074100c10 msgr2=0x7fd0741966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0737fe700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd074100c10 0x7fd0741966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0737fe700 1 -- 172.21.15.78:0/641715370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07419b6b0 con 0x7fd0741015d0 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0737fe700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd074196be0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd06800e9a0 tx=0x7fd06800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0717fa700 1 -- 172.21.15.78:0/641715370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06800ca10 con 0x7fd0741015d0 2024-03-16T11:05:50.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07419b9a0 con 0x7fd0741015d0 2024-03-16T11:05:50.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0717fa700 1 -- 172.21.15.78:0/641715370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd06800cbb0 con 0x7fd0741015d0 2024-03-16T11:05:50.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd0717fa700 1 -- 172.21.15.78:0/641715370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd068018680 con 0x7fd0741015d0 2024-03-16T11:05:50.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.614+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0741a3220 con 0x7fd0741015d0 2024-03-16T11:05:50.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.615+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd074068f90 con 0x7fd0741015d0 2024-03-16T11:05:50.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.616+0000 7fd0717fa700 1 -- 172.21.15.78:0/641715370 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd068010430 con 0x7fd0741015d0 2024-03-16T11:05:50.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.617+0000 7fd0717fa700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd05c0a2890 0x7fd05c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:50.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.617+0000 7fd073fff700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd05c0a2890 0x7fd05c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:50.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.617+0000 7fd0717fa700 1 -- 172.21.15.78:0/641715370 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd0680cee20 con 0x7fd0741015d0 2024-03-16T11:05:50.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.617+0000 7fd073fff700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd05c0a2890 0x7fd05c0a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd064009c60 tx=0x7fd064011040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.620+0000 7fd0717fa700 1 -- 172.21.15.78:0/641715370 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd06809bec0 con 0x7fd0741015d0 2024-03-16T11:05:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.861+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd07402db70 con 0x7fd0741015d0 2024-03-16T11:05:50.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.861+0000 7fd0717fa700 1 -- 172.21.15.78:0/641715370 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7fd06809b7c0 con 0x7fd0741015d0 2024-03-16T11:05:50.862 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd05c0a2890 msgr2=0x7fd05c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd05c0a2890 0x7fd05c0a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd064009c60 tx=0x7fd064011040 comp rx=0 tx=0).stop 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 msgr2=0x7fd074196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd074196be0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd06800e9a0 tx=0x7fd06800ecb0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 shutdown_connections 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd05c0a2890 0x7fd05c0a4d50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0741069f0 0x7fd07419af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd074100c10 0x7fd0741966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.863+0000 7fd07a283700 1 --2- 172.21.15.78:0/641715370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0741015d0 0x7fd074196be0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.864+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 >> 172.21.15.78:0/641715370 conn(0x7fd074076530 msgr2=0x7fd0740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.864+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 shutdown_connections 2024-03-16T11:05:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:50.864+0000 7fd07a283700 1 -- 172.21.15.78:0/641715370 wait complete. 2024-03-16T11:05:51.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:51 smithi078 ceph-mon[32235]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:51.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:51 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/641715370' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T11:05:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:51 smithi098 ceph-mon[35848]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:51 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/641715370' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T11:05:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:51 smithi153 ceph-mon[35521]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:05:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:51 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/641715370' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T11:05:51.689 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-16T11:05:51.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:51.690 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-03-16T11:05:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42090sec preferred_lft 42090sec 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-16T11:05:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-16T11:05:51.709 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T11:05:51.709 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:05:51.709 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-16T11:05:51.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-03-16T11:05:51.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-03-16T11:05:51.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-03-16T11:05:51.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T11:05:51.709 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-16T11:05:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:05:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:05:51.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:51.920 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T11:05:51.946 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-03-16T11:05:51.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:51.948 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-03-16T11:05:52.016 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-03-16T11:05:52.017 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T11:05:52.017 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-16T11:05:52.017 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-16T11:05:52.017 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T11:05:52.017 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:05:52.017 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-16T11:05:52.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-16T11:05:52.018 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-16T11:05:52.018 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-16T11:05:52.018 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-16T11:05:52.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:05:52.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:05:52.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:52.359 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-16T11:05:52.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:52.374 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-03-16T11:05:52.429 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:05:52.429 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-16T11:05:52.429 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T11:05:52.429 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-16T11:05:52.429 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-16T11:05:52.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:05:52.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:05:52.551 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-16T11:05:52.552 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-16T11:05:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:05:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:05:52.687 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:05:52.687 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-16T11:05:52.687 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-03-16T11:05:52.687 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T11:05:52.687 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-16T11:05:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:05:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:05:52.768 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-16T11:05:52.769 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T11:05:52.769 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-16T11:05:52.792 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-16T11:05:52.793 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T11:05:52.793 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-16T11:05:52.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-03-16T11:05:52.951 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T11:05:53.010 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-03-16T11:05:53.010 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-03-16T11:05:53.010 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-03-16T11:05:53.010 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-03-16T11:05:53.011 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-03-16T11:05:53.012 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/423aaf3f97e17b1d0cc1a5e5e051e9e2ba53f4272f504e98c8d7863e8506e480/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/fb1e8e3fd00a93af367d948525932bcbcc81edf80f277f86811cc963f874353c/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/0 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/26cc722b26f84079a7098fe9997b196f203186181bfb5d054409701fbe904409/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/ef20c091f21d7de64efe896d862b46d8cbc6d6d7a4e0bfda8f4a371dda535b4d/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/b3c4b210ea5d8c49071bae38fc56fb7641cc866bbc0838b8cec727b70e68fa11/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/2b1baf3368514f29dfd119893f2641f932dedc14b86c47776779f1880da608ad/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/618609b1a80cc24f311fa2a9cf028a3077a47eefa0261d7a96cf20ff62e9cd49/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/9027617660d0714387e5439cbc459c21940c431e0df8ee0c8cb3c45e6eb86b5e/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/f08ec27e25346589231af78b449b354ab6fe60d3dcdaf87fe339af67b1e29cb9/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/899e01c2739b0eb78ede970477dfe22716fbd9455d9e267e239e2312cc94de67/merged 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-03-16T11:05:53.013 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T11:05:53.014 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T11:05:53.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:53.014 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-03-16T11:05:53.069 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-16T11:05:53.070 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/31b1212f-132d-4bb4-a342-539dd397529d --client_fs=cephfs) 2024-03-16T11:05:53.120 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-03-16T11:05:53.151 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-03-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-03-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-03-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-03-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-03-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-03-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay 2024-03-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/423aaf3f97e17b1d0cc1a5e5e051e9e2ba53f4272f504e98c8d7863e8506e480/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/fb1e8e3fd00a93af367d948525932bcbcc81edf80f277f86811cc963f874353c/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/0 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/26cc722b26f84079a7098fe9997b196f203186181bfb5d054409701fbe904409/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/ef20c091f21d7de64efe896d862b46d8cbc6d6d7a4e0bfda8f4a371dda535b4d/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/b3c4b210ea5d8c49071bae38fc56fb7641cc866bbc0838b8cec727b70e68fa11/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/2b1baf3368514f29dfd119893f2641f932dedc14b86c47776779f1880da608ad/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/618609b1a80cc24f311fa2a9cf028a3077a47eefa0261d7a96cf20ff62e9cd49/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/9027617660d0714387e5439cbc459c21940c431e0df8ee0c8cb3c45e6eb86b5e/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/f08ec27e25346589231af78b449b354ab6fe60d3dcdaf87fe339af67b1e29cb9/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/899e01c2739b0eb78ede970477dfe22716fbd9455d9e267e239e2312cc94de67/merged 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-03-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T11:05:53.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:53.213 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-03-16T11:05:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:2024-03-16T11:05:53.215+0000 7f7dad037480 -1 init, newargv = 0x55a9e66cd4b0 newargc=15 2024-03-16T11:05:53.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:ceph-fuse[84740]: starting ceph client 2024-03-16T11:05:53.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:ceph-fuse[84740]: starting fuse 2024-03-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi078.stdout:119 2024-03-16T11:05:53.232 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-16T11:05:53.232 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-03-16T11:05:53.232 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-03-16T11:05:53.232 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-03-16T11:05:53.232 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-16T11:05:53.233 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.0")) 2024-03-16T11:05:53.234 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-03-16T11:05:53.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:53 smithi078 ceph-mon[32235]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:53.355 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.0.84740.asok 2024-03-16T11:05:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:05:53.366 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.84740.asok 2024-03-16T11:05:53.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:53.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.84740.asok status 2024-03-16T11:05:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:53 smithi098 ceph-mon[35848]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:53.504 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:05:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-03-16T11:05:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-16T11:05:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T11:05:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "0", 2024-03-16T11:05:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-03-16T11:05:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-16T11:05:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "84740", 2024-03-16T11:05:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/volumes/qa/sv_0/31b1212f-132d-4bb4-a342-539dd397529d" 2024-03-16T11:05:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:05:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-03-16T11:05:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-03-16T11:05:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 15291, 2024-03-16T11:05:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-03-16T11:05:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-03-16T11:05:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-03-16T11:05:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 15291 2024-03-16T11:05:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:05:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-03-16T11:05:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-16T11:05:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-03-16T11:05:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1764389319 2024-03-16T11:05:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:05:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:05:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-03-16T11:05:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-16T11:05:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-03-16T11:05:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1764389319 2024-03-16T11:05:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:05:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.15291 192.168.0.1:0/1764389319", 2024-03-16T11:05:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/1764389319", 2024-03-16T11:05:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 5, 2024-03-16T11:05:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 21, 2024-03-16T11:05:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 90, 2024-03-16T11:05:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-03-16T11:05:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-03-16T11:05:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-03-16T11:05:53.514 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:05:53.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:53.518 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-03-16T11:05:53.518 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-16T11:05:53.536 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-03-16T11:05:53.537 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-16T11:05:53.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:05:53.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-16T11:05:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:53 smithi153 ceph-mon[35521]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:05:53.607 INFO:teuthology.run_tasks:Running task exec... 2024-03-16T11:05:53.694 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T11:05:53.694 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T11:05:53.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 -- 172.21.15.78:0/3431974207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4100750 msgr2=0x7f4ef4100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/3431974207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4100750 0x7f4ef4100b70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009900 tx=0x7f4ee402bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 -- 172.21.15.78:0/3431974207 shutdown_connections 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/3431974207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4102300 0x7f4ef4108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/3431974207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef4101940 0x7f4ef4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/3431974207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4100750 0x7f4ef4100b70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 -- 172.21.15.78:0/3431974207 >> 172.21.15.78:0/3431974207 conn(0x7f4ef40fbcc0 msgr2=0x7f4ef40fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 -- 172.21.15.78:0/3431974207 shutdown_connections 2024-03-16T11:05:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.798+0000 7f4ef9be6640 1 -- 172.21.15.78:0/3431974207 wait complete. 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 Processor -- start 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 -- start start 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4100750 0x7f4ef4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4101940 0x7f4ef4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef4102300 0x7f4ef419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef410c270 con 0x7f4ef4101940 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef410c0f0 con 0x7f4ef4100750 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef9be6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef410c3f0 con 0x7f4ef4102300 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4101940 0x7f4ef4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4101940 0x7f4ef4197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36134/0 (socket says 172.21.15.78:36134) 2024-03-16T11:05:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 -- 172.21.15.78:0/2530729367 learned_addr learned my addr 172.21.15.78:0/2530729367 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef3fff640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef4102300 0x7f4ef419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef37fe640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4100750 0x7f4ef4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 -- 172.21.15.78:0/2530729367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef4102300 msgr2=0x7f4ef419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef4102300 0x7f4ef419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 -- 172.21.15.78:0/2530729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4100750 msgr2=0x7f4ef4197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4100750 0x7f4ef4197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 -- 172.21.15.78:0/2530729367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef419f600 con 0x7f4ef4101940 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.799+0000 7f4ef2ffd640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4101940 0x7f4ef4197f70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800e9f0 tx=0x7f4ee800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.800+0000 7f4ef0ff9640 1 -- 172.21.15.78:0/2530729367 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee800cd80 con 0x7f4ef4101940 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.800+0000 7f4ef0ff9640 1 -- 172.21.15.78:0/2530729367 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ee80054d0 con 0x7f4ef4101940 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.800+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef419f8f0 con 0x7f4ef4101940 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.800+0000 7f4ef0ff9640 1 -- 172.21.15.78:0/2530729367 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8010490 con 0x7f4ef4101940 2024-03-16T11:05:53.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.800+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef419fe40 con 0x7f4ef4101940 2024-03-16T11:05:53.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.801+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef41a00d0 con 0x7f4ef4101940 2024-03-16T11:05:53.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.802+0000 7f4ef0ff9640 1 -- 172.21.15.78:0/2530729367 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4ee8005020 con 0x7f4ef4101940 2024-03-16T11:05:53.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.802+0000 7f4ef0ff9640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4edc0a2920 0x7f4edc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:53.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.803+0000 7f4ef37fe640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4edc0a2920 0x7f4edc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:53.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.803+0000 7f4ef0ff9640 1 -- 172.21.15.78:0/2530729367 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f4ee80ced40 con 0x7f4ef4101940 2024-03-16T11:05:53.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.803+0000 7f4ef37fe640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4edc0a2920 0x7f4edc0a4de0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4005ab0 tx=0x7f4ee4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:53.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:53.807+0000 7f4ef0ff9640 1 -- 172.21.15.78:0/2530729367 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ee809bde0 con 0x7f4ef4101940 2024-03-16T11:05:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.020+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f4ef41a0300 con 0x7f4ef4101940 2024-03-16T11:05:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.080+0000 7f4ef0ff9640 1 -- 172.21.15.78:0/2530729367 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v22) ==== 118+0+0 (secure 0 0 0) 0x7f4ee809b6e0 con 0x7f4ef4101940 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4edc0a2920 msgr2=0x7f4edc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4edc0a2920 0x7f4edc0a4de0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4005ab0 tx=0x7f4ee4005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4101940 msgr2=0x7f4ef4197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4101940 0x7f4ef4197f70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800e9f0 tx=0x7f4ee800edf0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 shutdown_connections 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4edc0a2920 0x7f4edc0a4de0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef4102300 0x7f4ef419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ef4101940 0x7f4ef4197f70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 --2- 172.21.15.78:0/2530729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4100750 0x7f4ef4197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 >> 172.21.15.78:0/2530729367 conn(0x7f4ef40fbcc0 msgr2=0x7f4ef4069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 shutdown_connections 2024-03-16T11:05:54.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.082+0000 7f4ef9be6640 1 -- 172.21.15.78:0/2530729367 wait complete. 2024-03-16T11:05:54.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-16T11:05:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 -- 172.21.15.78:0/2549429808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 msgr2=0x7f6334100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 --2- 172.21.15.78:0/2549429808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f6334100b50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f632c009900 tx=0x7f632c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 -- 172.21.15.78:0/2549429808 shutdown_connections 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 --2- 172.21.15.78:0/2549429808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63341022e0 0x7f6334108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 --2- 172.21.15.78:0/2549429808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6334101920 0x7f6334101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 --2- 172.21.15.78:0/2549429808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f6334100b50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 -- 172.21.15.78:0/2549429808 >> 172.21.15.78:0/2549429808 conn(0x7f63340fbc80 msgr2=0x7f63340fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 -- 172.21.15.78:0/2549429808 shutdown_connections 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 -- 172.21.15.78:0/2549429808 wait complete. 2024-03-16T11:05:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 Processor -- start 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.197+0000 7f633c73b640 1 -- start start 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f633410adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6334101920 0x7f63341093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63341022e0 0x7f6334109930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f633410d330 con 0x7f6334100730 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f633410d1b0 con 0x7f6334101920 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f633410d4b0 con 0x7f63341022e0 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f633410adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f633410adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36156/0 (socket says 172.21.15.78:36156) 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 -- 172.21.15.78:0/980808660 learned_addr learned my addr 172.21.15.78:0/980808660 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f6339cd6640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6334101920 0x7f63341093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633acd8640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63341022e0 0x7f6334109930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 -- 172.21.15.78:0/980808660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63341022e0 msgr2=0x7f6334109930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63341022e0 0x7f6334109930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 -- 172.21.15.78:0/980808660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6334101920 msgr2=0x7f63341093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6334101920 0x7f63341093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 -- 172.21.15.78:0/980808660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633410a1c0 con 0x7f6334100730 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633a4d7640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f633410adb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f632c000c00 tx=0x7f632c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f63337fe640 1 -- 172.21.15.78:0/980808660 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c043020 con 0x7f6334100730 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f63337fe640 1 -- 172.21.15.78:0/980808660 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f632c004aa0 con 0x7f6334100730 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f63337fe640 1 -- 172.21.15.78:0/980808660 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c003d00 con 0x7f6334100730 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63341a1af0 con 0x7f6334100730 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.198+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63341a2090 con 0x7f6334100730 2024-03-16T11:05:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.199+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6334107ad0 con 0x7f6334100730 2024-03-16T11:05:54.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.200+0000 7f63337fe640 1 -- 172.21.15.78:0/980808660 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f632c046020 con 0x7f6334100730 2024-03-16T11:05:54.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.201+0000 7f63337fe640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f631c0a2920 0x7f631c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.201+0000 7f6339cd6640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f631c0a2920 0x7f631c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.201+0000 7f63337fe640 1 -- 172.21.15.78:0/980808660 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f632c0f0f40 con 0x7f6334100730 2024-03-16T11:05:54.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.201+0000 7f6339cd6640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f631c0a2920 0x7f631c0a4de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f633410ab20 tx=0x7f6328009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:54.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.205+0000 7f63337fe640 1 -- 172.21.15.78:0/980808660 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f632c04d020 con 0x7f6334100730 2024-03-16T11:05:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.407+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f6334061d80 con 0x7f631c0a2920 2024-03-16T11:05:54.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:54 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2530729367' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T11:05:54.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.423+0000 7f63337fe640 1 -- 172.21.15.78:0/980808660 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f6334061d80 con 0x7f631c0a2920 2024-03-16T11:05:54.423 INFO:teuthology.orchestra.run.smithi078.stdout:{} 2024-03-16T11:05:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f631c0a2920 msgr2=0x7f631c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f631c0a2920 0x7f631c0a4de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f633410ab20 tx=0x7f6328009680 comp rx=0 tx=0).stop 2024-03-16T11:05:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 msgr2=0x7f633410adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f633410adb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f632c000c00 tx=0x7f632c004270 comp rx=0 tx=0).stop 2024-03-16T11:05:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 shutdown_connections 2024-03-16T11:05:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f631c0a2920 0x7f631c0a4de0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63341022e0 0x7f6334109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6334101920 0x7f63341093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 --2- 172.21.15.78:0/980808660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334100730 0x7f633410adb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 >> 172.21.15.78:0/980808660 conn(0x7f63340fbc80 msgr2=0x7f6334104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:54.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.425+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 shutdown_connections 2024-03-16T11:05:54.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.426+0000 7f633c73b640 1 -- 172.21.15.78:0/980808660 wait complete. 2024-03-16T11:05:54.436 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-16T11:05:54.448 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-16T11:05:54.464 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:54 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2530729367' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T11:05:54.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:54 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2530729367' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T11:05:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 -- 172.21.15.78:0/1354764252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e41012b0 msgr2=0x7fa8e4108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 --2- 172.21.15.78:0/1354764252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e41012b0 0x7fa8e4108b70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009860 tx=0x7fa8d402bb70 comp rx=0 tx=0).stop 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 -- 172.21.15.78:0/1354764252 shutdown_connections 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 --2- 172.21.15.78:0/1354764252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e41012b0 0x7fa8e4108b70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 --2- 172.21.15.78:0/1354764252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e4100910 0x7fa8e4100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 --2- 172.21.15.78:0/1354764252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e4101cc0 0x7fa8e41020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 -- 172.21.15.78:0/1354764252 >> 172.21.15.78:0/1354764252 conn(0x7fa8e40fb400 msgr2=0x7fa8e40fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 -- 172.21.15.78:0/1354764252 shutdown_connections 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 -- 172.21.15.78:0/1354764252 wait complete. 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.567+0000 7fa8eac36640 1 Processor -- start 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 -- start start 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e4100910 0x7fa8e4100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e41012b0 0x7fa8e419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e4101cc0 0x7fa8e419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e410c210 con 0x7fa8e4100910 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e410c090 con 0x7fa8e41012b0 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e410c390 con 0x7fa8e4101cc0 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e4100910 0x7fa8e4100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e4100910 0x7fa8e4100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36182/0 (socket says 172.21.15.78:36182) 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 -- 172.21.15.78:0/3381685391 learned_addr learned my addr 172.21.15.78:0/3381685391 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e3fff640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e41012b0 0x7fa8e419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e91d3640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e4101cc0 0x7fa8e419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 -- 172.21.15.78:0/3381685391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e4101cc0 msgr2=0x7fa8e419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e4101cc0 0x7fa8e419ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 -- 172.21.15.78:0/3381685391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e41012b0 msgr2=0x7fa8e419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e41012b0 0x7fa8e419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 -- 172.21.15.78:0/3381685391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e419f5f0 con 0x7fa8e4100910 2024-03-16T11:05:54.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e89d2640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e4100910 0x7fa8e4100390 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00c420 tx=0x7fa8dc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:54.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e1ffb640 1 -- 172.21.15.78:0/3381685391 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc003970 con 0x7fa8e4100910 2024-03-16T11:05:54.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e1ffb640 1 -- 172.21.15.78:0/3381685391 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8dc004020 con 0x7fa8e4100910 2024-03-16T11:05:54.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e419f8e0 con 0x7fa8e4100910 2024-03-16T11:05:54.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.568+0000 7fa8e1ffb640 1 -- 172.21.15.78:0/3381685391 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00b980 con 0x7fa8e4100910 2024-03-16T11:05:54.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.569+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e419fe30 con 0x7fa8e4100910 2024-03-16T11:05:54.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.570+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e4107a50 con 0x7fa8e4100910 2024-03-16T11:05:54.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.571+0000 7fa8e1ffb640 1 -- 172.21.15.78:0/3381685391 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa8dc01d020 con 0x7fa8e4100910 2024-03-16T11:05:54.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.571+0000 7fa8e1ffb640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8cc0a2800 0x7fa8cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.571+0000 7fa8e3fff640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8cc0a2800 0x7fa8cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.571+0000 7fa8e1ffb640 1 -- 172.21.15.78:0/3381685391 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fa8dc0ce880 con 0x7fa8e4100910 2024-03-16T11:05:54.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.572+0000 7fa8e3fff640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8cc0a2800 0x7fa8cc0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4100100 tx=0x7fa8d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:54.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.575+0000 7fa8e1ffb640 1 -- 172.21.15.78:0/3381685391 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa8dc09b920 con 0x7fa8e4100910 2024-03-16T11:05:54.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.752+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8e40623e0 con 0x7fa8e4100910 2024-03-16T11:05:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.754+0000 7fa8e1ffb640 1 -- 172.21.15.78:0/3381685391 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7fa8dc09b220 con 0x7fa8e4100910 2024-03-16T11:05:54.754 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:54.754 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8cc0a2800 msgr2=0x7fa8cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8cc0a2800 0x7fa8cc0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4100100 tx=0x7fa8d8005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e4100910 msgr2=0x7fa8e4100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e4100910 0x7fa8e4100390 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00c420 tx=0x7fa8dc009f90 comp rx=0 tx=0).stop 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 shutdown_connections 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fa8cc0a2800 0x7fa8cc0a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e4101cc0 0x7fa8e419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8e41012b0 0x7fa8e419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 --2- 172.21.15.78:0/3381685391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8e4100910 0x7fa8e4100390 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 >> 172.21.15.78:0/3381685391 conn(0x7fa8e40fb400 msgr2=0x7fa8e4104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 shutdown_connections 2024-03-16T11:05:54.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.756+0000 7fa8eac36640 1 -- 172.21.15.78:0/3381685391 wait complete. 2024-03-16T11:05:54.765 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:05:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:05:54.412+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:05:54.766 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:05:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:05:54.412+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:05:54.766 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:05:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:05:54.412+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:05:54.766 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:05:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:05:54.412+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:05:54.766 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:05:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:05:54.412+0000 7f2d5ac72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:05:54.767 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:05:54.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 -- 172.21.15.78:0/38617369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8401006e0 msgr2=0x7fb840100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 --2- 172.21.15.78:0/38617369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8401006e0 0x7fb840100b00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb830009ca0 tx=0x7fb83002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 -- 172.21.15.78:0/38617369 shutdown_connections 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 --2- 172.21.15.78:0/38617369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb840102290 0x7fb840108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 --2- 172.21.15.78:0/38617369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401018d0 0x7fb840101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 --2- 172.21.15.78:0/38617369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8401006e0 0x7fb840100b00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 -- 172.21.15.78:0/38617369 >> 172.21.15.78:0/38617369 conn(0x7fb8400fbc70 msgr2=0x7fb8400fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:54.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 -- 172.21.15.78:0/38617369 shutdown_connections 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.872+0000 7fb845537640 1 -- 172.21.15.78:0/38617369 wait complete. 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 Processor -- start 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 -- start start 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401006e0 0x7fb8400694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8401018d0 0x7fb84019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb840102290 0x7fb84019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb84010c1e0 con 0x7fb840102290 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb84010c060 con 0x7fb8401006e0 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb845537640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb84010c360 con 0x7fb8401018d0 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb83f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb840102290 0x7fb84019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb83effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401006e0 0x7fb8400694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.873+0000 7fb83effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401006e0 0x7fb8400694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:53626/0 (socket says 172.21.15.78:53626) 2024-03-16T11:05:54.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83effd640 1 -- 172.21.15.78:0/2221268717 learned_addr learned my addr 172.21.15.78:0/2221268717 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83e7fc640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8401018d0 0x7fb84019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83effd640 1 -- 172.21.15.78:0/2221268717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8401018d0 msgr2=0x7fb84019c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83effd640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8401018d0 0x7fb84019c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83effd640 1 -- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb840102290 msgr2=0x7fb84019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83effd640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb840102290 0x7fb84019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83effd640 1 -- 172.21.15.78:0/2221268717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84019f6d0 con 0x7fb8401006e0 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83f7fe640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb840102290 0x7fb84019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb83effd640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401006e0 0x7fb8400694d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb830032380 tx=0x7fb830032820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb827fff640 1 -- 172.21.15.78:0/2221268717 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb830041020 con 0x7fb8401006e0 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb827fff640 1 -- 172.21.15.78:0/2221268717 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb830003e20 con 0x7fb8401006e0 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb84019f960 con 0x7fb8401006e0 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb827fff640 1 -- 172.21.15.78:0/2221268717 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb830039040 con 0x7fb8401006e0 2024-03-16T11:05:54.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.874+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb84019fe50 con 0x7fb8401006e0 2024-03-16T11:05:54.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.875+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb840107bc0 con 0x7fb8401006e0 2024-03-16T11:05:54.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.876+0000 7fb827fff640 1 -- 172.21.15.78:0/2221268717 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb830044020 con 0x7fb8401006e0 2024-03-16T11:05:54.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.877+0000 7fb827fff640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8280a2800 0x7fb8280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:54.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.877+0000 7fb83e7fc640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8280a2800 0x7fb8280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:54.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.877+0000 7fb827fff640 1 -- 172.21.15.78:0/2221268717 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb8300f0e70 con 0x7fb8401006e0 2024-03-16T11:05:54.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.877+0000 7fb83e7fc640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8280a2800 0x7fb8280a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb840069240 tx=0x7fb834005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:54.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:54.881+0000 7fb827fff640 1 -- 172.21.15.78:0/2221268717 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb83004b020 con 0x7fb8401006e0 2024-03-16T11:05:55.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.055+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb84002d340 con 0x7fb8401006e0 2024-03-16T11:05:55.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.055+0000 7fb827fff640 1 -- 172.21.15.78:0/2221268717 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fb8300bdf10 con 0x7fb8401006e0 2024-03-16T11:05:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8280a2800 msgr2=0x7fb8280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8280a2800 0x7fb8280a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb840069240 tx=0x7fb834005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401006e0 msgr2=0x7fb8400694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401006e0 0x7fb8400694d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb830032380 tx=0x7fb830032820 comp rx=0 tx=0).stop 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 shutdown_connections 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb840102290 0x7fb84019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8280a2800 0x7fb8280a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8401018d0 0x7fb84019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 --2- 172.21.15.78:0/2221268717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8401006e0 0x7fb8400694d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 >> 172.21.15.78:0/2221268717 conn(0x7fb8400fbc70 msgr2=0x7fb840104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:55.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 shutdown_connections 2024-03-16T11:05:55.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.057+0000 7fb845537640 1 -- 172.21.15.78:0/2221268717 wait complete. 2024-03-16T11:05:55.069 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:05:55.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 -- 172.21.15.78:0/251554767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 msgr2=0x7fc624108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 --2- 172.21.15.78:0/251554767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc624108b40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009ca0 tx=0x7fc61c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 -- 172.21.15.78:0/251554767 shutdown_connections 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 --2- 172.21.15.78:0/251554767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc624108b40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 --2- 172.21.15.78:0/251554767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6241018f0 0x7fc624101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 --2- 172.21.15.78:0/251554767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc624100700 0x7fc624100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 -- 172.21.15.78:0/251554767 >> 172.21.15.78:0/251554767 conn(0x7fc6240fbc90 msgr2=0x7fc6240fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 -- 172.21.15.78:0/251554767 shutdown_connections 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.176+0000 7fc62b197640 1 -- 172.21.15.78:0/251554767 wait complete. 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 Processor -- start 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 -- start start 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc624100700 0x7fc6241979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6241018f0 0x7fc624197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc62419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc62410c230 con 0x7fc6241022b0 2024-03-16T11:05:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc62410c0b0 con 0x7fc6241018f0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc62b197640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc62410c3b0 con 0x7fc624100700 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc62419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc628f33640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc624100700 0x7fc6241979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc62419ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36208/0 (socket says 172.21.15.78:36208) 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 -- 172.21.15.78:0/3454475969 learned_addr learned my addr 172.21.15.78:0/3454475969 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc61bfff640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6241018f0 0x7fc624197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 -- 172.21.15.78:0/3454475969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc624100700 msgr2=0x7fc6241979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc624100700 0x7fc6241979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 -- 172.21.15.78:0/3454475969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6241018f0 msgr2=0x7fc624197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6241018f0 0x7fc624197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 -- 172.21.15.78:0/3454475969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62419f5e0 con 0x7fc6241022b0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.177+0000 7fc629734640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc62419ef70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc61c02c700 tx=0x7fc61c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.178+0000 7fc619ffb640 1 -- 172.21.15.78:0/3454475969 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c041020 con 0x7fc6241022b0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.178+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62419f870 con 0x7fc6241022b0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.178+0000 7fc619ffb640 1 -- 172.21.15.78:0/3454475969 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc61c003e20 con 0x7fc6241022b0 2024-03-16T11:05:55.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.178+0000 7fc619ffb640 1 -- 172.21.15.78:0/3454475969 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c039040 con 0x7fc6241022b0 2024-03-16T11:05:55.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.178+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62419fd60 con 0x7fc6241022b0 2024-03-16T11:05:55.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.179+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc62419ffc0 con 0x7fc6241022b0 2024-03-16T11:05:55.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.180+0000 7fc619ffb640 1 -- 172.21.15.78:0/3454475969 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fc61c044020 con 0x7fc6241022b0 2024-03-16T11:05:55.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.180+0000 7fc619ffb640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc60c0a2890 0x7fc60c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.180+0000 7fc628f33640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc60c0a2890 0x7fc60c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.180+0000 7fc619ffb640 1 -- 172.21.15.78:0/3454475969 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc61c0f0ce0 con 0x7fc6241022b0 2024-03-16T11:05:55.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.181+0000 7fc628f33640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc60c0a2890 0x7fc60c0a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc624101750 tx=0x7fc620005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:55.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.185+0000 7fc619ffb640 1 -- 172.21.15.78:0/3454475969 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc61c04b020 con 0x7fc6241022b0 2024-03-16T11:05:55.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.360+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6241a0350 con 0x7fc6241022b0 2024-03-16T11:05:55.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.360+0000 7fc619ffb640 1 -- 172.21.15.78:0/3454475969 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fc61c0bdd30 con 0x7fc6241022b0 2024-03-16T11:05:55.361 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc60c0a2890 msgr2=0x7fc60c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc60c0a2890 0x7fc60c0a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc624101750 tx=0x7fc620005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 msgr2=0x7fc62419ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc62419ef70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc61c02c700 tx=0x7fc61c0328a0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 shutdown_connections 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6241022b0 0x7fc62419ef70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6241018f0 0x7fc624197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc60c0a2890 0x7fc60c0a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 --2- 172.21.15.78:0/3454475969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc624100700 0x7fc6241979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 >> 172.21.15.78:0/3454475969 conn(0x7fc6240fbc90 msgr2=0x7fc6240fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.362+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 shutdown_connections 2024-03-16T11:05:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.363+0000 7fc62b197640 1 -- 172.21.15.78:0/3454475969 wait complete. 2024-03-16T11:05:55.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:55 smithi078 ceph-mon[32235]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:05:55.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:55 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2530729367' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T11:05:55.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:55 smithi078 ceph-mon[32235]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:05:55.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:55 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3381685391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:55.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:55 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2221268717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:55.374 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:05:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:55 smithi098 ceph-mon[35848]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:05:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:55 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2530729367' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T11:05:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:55 smithi098 ceph-mon[35848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:05:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:55 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3381685391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:55.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:55 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2221268717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 -- 172.21.15.78:0/2081757183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd844100700 msgr2=0x7fd844100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 --2- 172.21.15.78:0/2081757183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd844100700 0x7fd844100b20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd834009900 tx=0x7fd83402bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 -- 172.21.15.78:0/2081757183 shutdown_connections 2024-03-16T11:05:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 --2- 172.21.15.78:0/2081757183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8441022b0 0x7fd844108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 --2- 172.21.15.78:0/2081757183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8441018f0 0x7fd844101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 --2- 172.21.15.78:0/2081757183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd844100700 0x7fd844100b20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 -- 172.21.15.78:0/2081757183 >> 172.21.15.78:0/2081757183 conn(0x7fd8440fbc90 msgr2=0x7fd8440fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 -- 172.21.15.78:0/2081757183 shutdown_connections 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.478+0000 7fd849636640 1 -- 172.21.15.78:0/2081757183 wait complete. 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 Processor -- start 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 -- start start 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd844100700 0x7fd844196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8441018f0 0x7fd844196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8441022b0 0x7fd84419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd84410c230 con 0x7fd8441018f0 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd84410c0b0 con 0x7fd844100700 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd849636640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd84410c3b0 con 0x7fd8441022b0 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8441018f0 0x7fd844196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd842ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd844100700 0x7fd844196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8441018f0 0x7fd844196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54660/0 (socket says 172.21.15.78:54660) 2024-03-16T11:05:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 -- 172.21.15.78:0/47625257 learned_addr learned my addr 172.21.15.78:0/47625257 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd8437fe640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8441022b0 0x7fd84419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 -- 172.21.15.78:0/47625257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8441022b0 msgr2=0x7fd84419df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8441022b0 0x7fd84419df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 -- 172.21.15.78:0/47625257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd844100700 msgr2=0x7fd844196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd844100700 0x7fd844196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd83bfff640 1 -- 172.21.15.78:0/47625257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84419e570 con 0x7fd8441018f0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.479+0000 7fd842ffd640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd844100700 0x7fd844196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.480+0000 7fd83bfff640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8441018f0 0x7fd844196e90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00e9f0 tx=0x7fd82c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.480+0000 7fd840ff9640 1 -- 172.21.15.78:0/47625257 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c00cd80 con 0x7fd8441018f0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.480+0000 7fd840ff9640 1 -- 172.21.15.78:0/47625257 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd82c0108e0 con 0x7fd8441018f0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.480+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84419e860 con 0x7fd8441018f0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.480+0000 7fd840ff9640 1 -- 172.21.15.78:0/47625257 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c018920 con 0x7fd8441018f0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.480+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd84419ee00 con 0x7fd8441018f0 2024-03-16T11:05:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.481+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd844107b40 con 0x7fd8441018f0 2024-03-16T11:05:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.482+0000 7fd840ff9640 1 -- 172.21.15.78:0/47625257 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fd82c010a90 con 0x7fd8441018f0 2024-03-16T11:05:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.482+0000 7fd840ff9640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd8300a2910 0x7fd8300a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.483+0000 7fd842ffd640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd8300a2910 0x7fd8300a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.483+0000 7fd840ff9640 1 -- 172.21.15.78:0/47625257 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd82c0cee80 con 0x7fd8441018f0 2024-03-16T11:05:55.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.483+0000 7fd842ffd640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd8300a2910 0x7fd8300a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd834005ab0 tx=0x7fd834005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:55.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.487+0000 7fd840ff9640 1 -- 172.21.15.78:0/47625257 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd82c09b770 con 0x7fd8441018f0 2024-03-16T11:05:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:55 smithi153 ceph-mon[35521]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:05:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:55 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2530729367' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T11:05:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:55 smithi153 ceph-mon[35521]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:05:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:55 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3381685391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:55 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2221268717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:55.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.659+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd84402d400 con 0x7fd8441018f0 2024-03-16T11:05:55.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.660+0000 7fd840ff9640 1 -- 172.21.15.78:0/47625257 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fd82c09f810 con 0x7fd8441018f0 2024-03-16T11:05:55.660 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:05:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.661+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd8300a2910 msgr2=0x7fd8300a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.661+0000 7fd849636640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd8300a2910 0x7fd8300a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd834005ab0 tx=0x7fd834005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.661+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8441018f0 msgr2=0x7fd844196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.661+0000 7fd849636640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8441018f0 0x7fd844196e90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00e9f0 tx=0x7fd82c00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.661+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 shutdown_connections 2024-03-16T11:05:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.661+0000 7fd849636640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd8300a2910 0x7fd8300a4dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.661+0000 7fd849636640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8441022b0 0x7fd84419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.662+0000 7fd849636640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8441018f0 0x7fd844196e90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.662+0000 7fd849636640 1 --2- 172.21.15.78:0/47625257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd844100700 0x7fd844196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.662+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 >> 172.21.15.78:0/47625257 conn(0x7fd8440fbc90 msgr2=0x7fd844104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.662+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 shutdown_connections 2024-03-16T11:05:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.662+0000 7fd849636640 1 -- 172.21.15.78:0/47625257 wait complete. 2024-03-16T11:05:55.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:55.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 -- 172.21.15.78:0/1310429760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c102e40 msgr2=0x7f012c103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 --2- 172.21.15.78:0/1310429760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c102e40 0x7f012c103260 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f011c009900 tx=0x7f011c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 -- 172.21.15.78:0/1310429760 shutdown_connections 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 --2- 172.21.15.78:0/1310429760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f012c100270 0x7f012c109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 --2- 172.21.15.78:0/1310429760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f012c0ff8b0 0x7f012c0ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 --2- 172.21.15.78:0/1310429760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c102e40 0x7f012c103260 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 -- 172.21.15.78:0/1310429760 >> 172.21.15.78:0/1310429760 conn(0x7f012c0fb410 msgr2=0x7f012c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 -- 172.21.15.78:0/1310429760 shutdown_connections 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 -- 172.21.15.78:0/1310429760 wait complete. 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.774+0000 7f013206a640 1 Processor -- start 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f013206a640 1 -- start start 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f013206a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f012c0ff8b0 0x7f012c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f013206a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c100270 0x7f012c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f013206a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f012c102e40 0x7f012c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f013206a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f012c10d340 con 0x7f012c100270 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f013206a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f012c10d1c0 con 0x7f012c0ff8b0 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f013206a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f012c10d4c0 con 0x7f012c102e40 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c100270 0x7f012c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c100270 0x7f012c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54676/0 (socket says 172.21.15.78:54676) 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 -- 172.21.15.78:0/420672461 learned_addr learned my addr 172.21.15.78:0/420672461 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012b7fe640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f012c0ff8b0 0x7f012c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012bfff640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f012c102e40 0x7f012c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 -- 172.21.15.78:0/420672461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f012c102e40 msgr2=0x7f012c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f012c102e40 0x7f012c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 -- 172.21.15.78:0/420672461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f012c0ff8b0 msgr2=0x7f012c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f012c0ff8b0 0x7f012c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 -- 172.21.15.78:0/420672461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c19f680 con 0x7f012c100270 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012b7fe640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f012c0ff8b0 0x7f012c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.775+0000 7f012affd640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c100270 0x7f012c197fa0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f012000e9f0 tx=0x7f012000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:55.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.776+0000 7f0128ff9640 1 -- 172.21.15.78:0/420672461 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012000cd80 con 0x7f012c100270 2024-03-16T11:05:55.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.776+0000 7f0128ff9640 1 -- 172.21.15.78:0/420672461 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01200054d0 con 0x7f012c100270 2024-03-16T11:05:55.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.776+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012c19f970 con 0x7f012c100270 2024-03-16T11:05:55.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.776+0000 7f0128ff9640 1 -- 172.21.15.78:0/420672461 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120010490 con 0x7f012c100270 2024-03-16T11:05:55.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.776+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012c19fe70 con 0x7f012c100270 2024-03-16T11:05:55.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.777+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f012c108e20 con 0x7f012c100270 2024-03-16T11:05:55.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.778+0000 7f0128ff9640 1 -- 172.21.15.78:0/420672461 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f0120005000 con 0x7f012c100270 2024-03-16T11:05:55.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.778+0000 7f0128ff9640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01140a2890 0x7f01140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:55.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.778+0000 7f012b7fe640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01140a2890 0x7f01140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:55.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.778+0000 7f0128ff9640 1 -- 172.21.15.78:0/420672461 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f01200cf150 con 0x7f012c100270 2024-03-16T11:05:55.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.779+0000 7f012b7fe640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01140a2890 0x7f01140a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f011c005ab0 tx=0x7f011c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:55.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.782+0000 7f0128ff9640 1 -- 172.21.15.78:0/420672461 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0120053740 con 0x7f012c100270 2024-03-16T11:05:55.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.956+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f012c02d1b0 con 0x7f012c100270 2024-03-16T11:05:55.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.957+0000 7f0128ff9640 1 -- 172.21.15.78:0/420672461 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f012c02d1b0 con 0x7f012c100270 2024-03-16T11:05:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01140a2890 msgr2=0x7f01140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01140a2890 0x7f01140a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f011c005ab0 tx=0x7f011c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c100270 msgr2=0x7f012c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c100270 0x7f012c197fa0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f012000e9f0 tx=0x7f012000edf0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 shutdown_connections 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01140a2890 0x7f01140a4d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f012c102e40 0x7f012c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f012c100270 0x7f012c197fa0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 --2- 172.21.15.78:0/420672461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f012c0ff8b0 0x7f012c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 >> 172.21.15.78:0/420672461 conn(0x7f012c0fb410 msgr2=0x7f012c105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 shutdown_connections 2024-03-16T11:05:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:55.959+0000 7f013206a640 1 -- 172.21.15.78:0/420672461 wait complete. 2024-03-16T11:05:55.971 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:05:55.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:05:56.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 -- 172.21.15.78:0/4219875958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c100700 msgr2=0x7ff52c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 --2- 172.21.15.78:0/4219875958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c100700 0x7ff52c100b20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff524009ca0 tx=0x7ff52402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 -- 172.21.15.78:0/4219875958 shutdown_connections 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 --2- 172.21.15.78:0/4219875958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c1022b0 0x7ff52c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 --2- 172.21.15.78:0/4219875958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c1018f0 0x7ff52c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 --2- 172.21.15.78:0/4219875958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c100700 0x7ff52c100b20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 -- 172.21.15.78:0/4219875958 >> 172.21.15.78:0/4219875958 conn(0x7ff52c0fbc90 msgr2=0x7ff52c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 -- 172.21.15.78:0/4219875958 shutdown_connections 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 -- 172.21.15.78:0/4219875958 wait complete. 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 Processor -- start 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.076+0000 7ff53428c640 1 -- start start 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c100700 0x7ff52c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c1018f0 0x7ff52c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c1022b0 0x7ff52c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c10d340 con 0x7ff52c1022b0 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c10d1c0 con 0x7ff52c100700 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c10d4c0 con 0x7ff52c1018f0 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c1022b0 0x7ff52c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c1022b0 0x7ff52c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54702/0 (socket says 172.21.15.78:54702) 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532028640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c100700 0x7ff52c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff531827640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c1018f0 0x7ff52c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 -- 172.21.15.78:0/3293348003 learned_addr learned my addr 172.21.15.78:0/3293348003 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 -- 172.21.15.78:0/3293348003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c1018f0 msgr2=0x7ff52c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c1018f0 0x7ff52c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 -- 172.21.15.78:0/3293348003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c100700 msgr2=0x7ff52c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c100700 0x7ff52c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 -- 172.21.15.78:0/3293348003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c19f670 con 0x7ff52c1022b0 2024-03-16T11:05:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff531827640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c1018f0 0x7ff52c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532028640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c100700 0x7ff52c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff532829640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c1022b0 0x7ff52c19f000 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e970 tx=0x7ff52800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff5237fe640 1 -- 172.21.15.78:0/3293348003 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52800ccc0 con 0x7ff52c1022b0 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff5237fe640 1 -- 172.21.15.78:0/3293348003 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff52800ce60 con 0x7ff52c1022b0 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c19f960 con 0x7ff52c1022b0 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff5237fe640 1 -- 172.21.15.78:0/3293348003 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528010430 con 0x7ff52c1022b0 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.077+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c19feb0 con 0x7ff52c1022b0 2024-03-16T11:05:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.078+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c107b90 con 0x7ff52c1022b0 2024-03-16T11:05:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.079+0000 7ff5237fe640 1 -- 172.21.15.78:0/3293348003 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7ff528004f90 con 0x7ff52c1022b0 2024-03-16T11:05:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.080+0000 7ff5237fe640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff5180a2890 0x7ff5180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.080+0000 7ff532028640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff5180a2890 0x7ff5180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.080+0000 7ff5237fe640 1 -- 172.21.15.78:0/3293348003 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff5280cef90 con 0x7ff52c1022b0 2024-03-16T11:05:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.080+0000 7ff532028640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff5180a2890 0x7ff5180a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff524032380 tx=0x7ff524000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.084+0000 7ff5237fe640 1 -- 172.21.15.78:0/3293348003 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff52809bed0 con 0x7ff52c1022b0 2024-03-16T11:05:56.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.262+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff52c02d400 con 0x7ff52c1022b0 2024-03-16T11:05:56.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.262+0000 7ff5237fe640 1 -- 172.21.15.78:0/3293348003 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7ff52809b7d0 con 0x7ff52c1022b0 2024-03-16T11:05:56.263 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff5180a2890 msgr2=0x7ff5180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff5180a2890 0x7ff5180a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff524032380 tx=0x7ff524000f40 comp rx=0 tx=0).stop 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c1022b0 msgr2=0x7ff52c19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c1022b0 0x7ff52c19f000 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e970 tx=0x7ff52800ed70 comp rx=0 tx=0).stop 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 shutdown_connections 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c1022b0 0x7ff52c19f000 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff5180a2890 0x7ff5180a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c1018f0 0x7ff52c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 --2- 172.21.15.78:0/3293348003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c100700 0x7ff52c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 >> 172.21.15.78:0/3293348003 conn(0x7ff52c0fbc90 msgr2=0x7ff52c104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 shutdown_connections 2024-03-16T11:05:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.264+0000 7ff53428c640 1 -- 172.21.15.78:0/3293348003 wait complete. 2024-03-16T11:05:56.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:56 smithi078 ceph-mon[32235]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:56 smithi078 ceph-mon[32235]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-03-16T11:05:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:56 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3454475969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:56 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/47625257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:56 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/420672461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:56.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 -- 172.21.15.78:0/4129626041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741006f0 msgr2=0x7f1774100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 --2- 172.21.15.78:0/4129626041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741006f0 0x7f1774100b10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1764009900 tx=0x7f176402bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 -- 172.21.15.78:0/4129626041 shutdown_connections 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 --2- 172.21.15.78:0/4129626041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17741022a0 0x7f1774108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 --2- 172.21.15.78:0/4129626041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17741018e0 0x7f1774101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 --2- 172.21.15.78:0/4129626041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741006f0 0x7f1774100b10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 -- 172.21.15.78:0/4129626041 >> 172.21.15.78:0/4129626041 conn(0x7f17740fbc80 msgr2=0x7f17740fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 -- 172.21.15.78:0/4129626041 shutdown_connections 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 -- 172.21.15.78:0/4129626041 wait complete. 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 Processor -- start 2024-03-16T11:05:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.381+0000 7f177a5d4640 1 -- start start 2024-03-16T11:05:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f177a5d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17741006f0 0x7f1774197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f177a5d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17741018e0 0x7f1774197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f177a5d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741022a0 0x7f177419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f177a5d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f177410c210 con 0x7f17741022a0 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f177a5d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f177410c090 con 0x7f17741018e0 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f177a5d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f177410c390 con 0x7f17741006f0 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741022a0 0x7f177419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1773fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17741006f0 0x7f1774197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741022a0 0x7f177419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54710/0 (socket says 172.21.15.78:54710) 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 -- 172.21.15.78:0/354591086 learned_addr learned my addr 172.21.15.78:0/354591086 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f17737fe640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17741018e0 0x7f1774197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 -- 172.21.15.78:0/354591086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17741006f0 msgr2=0x7f1774197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17741006f0 0x7f1774197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 -- 172.21.15.78:0/354591086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17741018e0 msgr2=0x7f1774197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17741018e0 0x7f1774197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 -- 172.21.15.78:0/354591086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177419f6c0 con 0x7f17741022a0 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f17737fe640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17741018e0 0x7f1774197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:56.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f1778b71640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741022a0 0x7f177419f050 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f176c00eb10 tx=0x7f176c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.382+0000 7f17717fa640 1 -- 172.21.15.78:0/354591086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c004b20 con 0x7f17741022a0 2024-03-16T11:05:56.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.383+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177419f9b0 con 0x7f17741022a0 2024-03-16T11:05:56.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.383+0000 7f17717fa640 1 -- 172.21.15.78:0/354591086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f176c004cc0 con 0x7f17741022a0 2024-03-16T11:05:56.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.383+0000 7f17717fa640 1 -- 172.21.15.78:0/354591086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c010690 con 0x7f17741022a0 2024-03-16T11:05:56.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.383+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177419ff00 con 0x7f17741022a0 2024-03-16T11:05:56.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.383+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17741a0190 con 0x7f17741022a0 2024-03-16T11:05:56.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.385+0000 7f17717fa640 1 -- 172.21.15.78:0/354591086 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f176c005000 con 0x7f17741022a0 2024-03-16T11:05:56.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.385+0000 7f17717fa640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f175c0a2890 0x7f175c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.385+0000 7f1773fff640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f175c0a2890 0x7f175c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.385+0000 7f17717fa640 1 -- 172.21.15.78:0/354591086 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f176c0cf340 con 0x7f17741022a0 2024-03-16T11:05:56.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.386+0000 7f1773fff640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f175c0a2890 0x7f175c0a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1764005ab0 tx=0x7f1764005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.389+0000 7f17717fa640 1 -- 172.21.15.78:0/354591086 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f176c053740 con 0x7f17741022a0 2024-03-16T11:05:56.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:56 smithi098 ceph-mon[35848]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:56.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:56 smithi098 ceph-mon[35848]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-03-16T11:05:56.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:56 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3454475969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:56 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/47625257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:56.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:56 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/420672461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:56.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:56 smithi153 ceph-mon[35521]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:05:56.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:56 smithi153 ceph-mon[35521]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-03-16T11:05:56.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:56 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3454475969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:56.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:56 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/47625257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:56.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:56 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/420672461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.562+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f17741a0360 con 0x7f17741022a0 2024-03-16T11:05:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.563+0000 7f17717fa640 1 -- 172.21.15.78:0/354591086 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f176c09c280 con 0x7f17741022a0 2024-03-16T11:05:56.563 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:05:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f175c0a2890 msgr2=0x7f175c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f175c0a2890 0x7f175c0a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1764005ab0 tx=0x7f1764005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741022a0 msgr2=0x7f177419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741022a0 0x7f177419f050 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f176c00eb10 tx=0x7f176c00c2d0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 shutdown_connections 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17741022a0 0x7f177419f050 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17741018e0 0x7f1774197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f175c0a2890 0x7f175c0a4d50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 --2- 172.21.15.78:0/354591086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17741006f0 0x7f1774197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 >> 172.21.15.78:0/354591086 conn(0x7f17740fbc80 msgr2=0x7f1774069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 shutdown_connections 2024-03-16T11:05:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.565+0000 7f177a5d4640 1 -- 172.21.15.78:0/354591086 wait complete. 2024-03-16T11:05:56.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:05:56.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 -- 172.21.15.78:0/3351026671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e901018d0 msgr2=0x7f8e90101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 --2- 172.21.15.78:0/3351026671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e901018d0 0x7f8e90101d50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009900 tx=0x7f8e8402bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:56.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 -- 172.21.15.78:0/3351026671 shutdown_connections 2024-03-16T11:05:56.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 --2- 172.21.15.78:0/3351026671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e90102290 0x7f8e90108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 --2- 172.21.15.78:0/3351026671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e901018d0 0x7f8e90101d50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 --2- 172.21.15.78:0/3351026671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e901006e0 0x7f8e90100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 -- 172.21.15.78:0/3351026671 >> 172.21.15.78:0/3351026671 conn(0x7f8e900fbc70 msgr2=0x7f8e900fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.679+0000 7f8e95c81640 1 -- 172.21.15.78:0/3351026671 shutdown_connections 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 -- 172.21.15.78:0/3351026671 wait complete. 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 Processor -- start 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 -- start start 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e901006e0 0x7f8e90197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e901018d0 0x7f8e90197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e90102290 0x7f8e9019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9010d330 con 0x7f8e90102290 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9010d1b0 con 0x7f8e901006e0 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e95c81640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9010d4b0 con 0x7f8e901018d0 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.680+0000 7f8e8ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e90102290 0x7f8e9019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e901018d0 0x7f8e90197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e90102290 0x7f8e9019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54738/0 (socket says 172.21.15.78:54738) 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e901006e0 0x7f8e90197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e901018d0 0x7f8e90197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:41058/0 (socket says 172.21.15.78:41058) 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 -- 172.21.15.78:0/768457151 learned_addr learned my addr 172.21.15.78:0/768457151 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 -- 172.21.15.78:0/768457151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e901018d0 msgr2=0x7f8e90197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e901018d0 0x7f8e90197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 -- 172.21.15.78:0/768457151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e901006e0 msgr2=0x7f8e90197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e901006e0 0x7f8e90197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 -- 172.21.15.78:0/768457151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9019f6a0 con 0x7f8e90102290 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8f7fe640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e901006e0 0x7f8e90197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8ffff640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e90102290 0x7f8e9019f030 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800efd0 tx=0x7f8e8800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8cff9640 1 -- 172.21.15.78:0/768457151 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88003b70 con 0x7f8e90102290 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8cff9640 1 -- 172.21.15.78:0/768457151 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e88003d10 con 0x7f8e90102290 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9019f990 con 0x7f8e90102290 2024-03-16T11:05:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e8cff9640 1 -- 172.21.15.78:0/768457151 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88010980 con 0x7f8e90102290 2024-03-16T11:05:56.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.681+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9019fe60 con 0x7f8e90102290 2024-03-16T11:05:56.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.682+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e90107b10 con 0x7f8e90102290 2024-03-16T11:05:56.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.683+0000 7f8e8cff9640 1 -- 172.21.15.78:0/768457151 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f8e88004020 con 0x7f8e90102290 2024-03-16T11:05:56.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.684+0000 7f8e8cff9640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e780a2890 0x7f8e780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.684+0000 7f8e8f7fe640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e780a2890 0x7f8e780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.684+0000 7f8e8cff9640 1 -- 172.21.15.78:0/768457151 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8e880cef90 con 0x7f8e90102290 2024-03-16T11:05:56.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.684+0000 7f8e8f7fe640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e780a2890 0x7f8e780a4d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8e90101730 tx=0x7f8e80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.688+0000 7f8e8cff9640 1 -- 172.21.15.78:0/768457151 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e8809bf90 con 0x7f8e90102290 2024-03-16T11:05:56.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.861+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8e9002d200 con 0x7f8e90102290 2024-03-16T11:05:56.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.862+0000 7f8e8cff9640 1 -- 172.21.15.78:0/768457151 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f8e88084e20 con 0x7f8e90102290 2024-03-16T11:05:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:05:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e780a2890 msgr2=0x7f8e780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e780a2890 0x7f8e780a4d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8e90101730 tx=0x7f8e80005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e90102290 msgr2=0x7f8e9019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e90102290 0x7f8e9019f030 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800efd0 tx=0x7f8e8800c6a0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 shutdown_connections 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e90102290 0x7f8e9019f030 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8e780a2890 0x7f8e780a4d50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e901018d0 0x7f8e90197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 --2- 172.21.15.78:0/768457151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e901006e0 0x7f8e90197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 >> 172.21.15.78:0/768457151 conn(0x7f8e900fbc70 msgr2=0x7f8e90104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 shutdown_connections 2024-03-16T11:05:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.864+0000 7f8e95c81640 1 -- 172.21.15.78:0/768457151 wait complete. 2024-03-16T11:05:56.875 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-16T11:05:56.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 -- 172.21.15.78:0/2697444266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 msgr2=0x7ff5b0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/2697444266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0101c20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac009900 tx=0x7ff5ac02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 -- 172.21.15.78:0/2697444266 shutdown_connections 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/2697444266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b01033b0 0x7ff5b0109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/2697444266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5b01029f0 0x7ff5b0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/2697444266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0101c20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 -- 172.21.15.78:0/2697444266 >> 172.21.15.78:0/2697444266 conn(0x7ff5b00fcd90 msgr2=0x7ff5b00ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 -- 172.21.15.78:0/2697444266 shutdown_connections 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.976+0000 7ff5b80f1640 1 -- 172.21.15.78:0/2697444266 wait complete. 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 Processor -- start 2024-03-16T11:05:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 -- start start 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b01029f0 0x7ff5b0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5b01033b0 0x7ff5b019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b010d320 con 0x7ff5b0101800 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b010d1a0 con 0x7ff5b01029f0 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b80f1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b010d4a0 con 0x7ff5b01033b0 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54762/0 (socket says 172.21.15.78:54762) 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 -- 172.21.15.78:0/3011666219 learned_addr learned my addr 172.21.15.78:0/3011666219 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b568c640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b01029f0 0x7ff5b0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b668e640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5b01033b0 0x7ff5b019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 -- 172.21.15.78:0/3011666219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5b01033b0 msgr2=0x7ff5b019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5b01033b0 0x7ff5b019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 -- 172.21.15.78:0/3011666219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b01029f0 msgr2=0x7ff5b0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b01029f0 0x7ff5b0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 -- 172.21.15.78:0/3011666219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b019f680 con 0x7ff5b0101800 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.977+0000 7ff5b5e8d640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0197a60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac000c00 tx=0x7ff5ac005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.978+0000 7ff5a6ffd640 1 -- 172.21.15.78:0/3011666219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac043020 con 0x7ff5b0101800 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.978+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b019f910 con 0x7ff5b0101800 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.978+0000 7ff5a6ffd640 1 -- 172.21.15.78:0/3011666219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5ac004350 con 0x7ff5b0101800 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.978+0000 7ff5a6ffd640 1 -- 172.21.15.78:0/3011666219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac003b10 con 0x7ff5b0101800 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.978+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b019feb0 con 0x7ff5b0101800 2024-03-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.979+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b0068a20 con 0x7ff5b0101800 2024-03-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.980+0000 7ff5a6ffd640 1 -- 172.21.15.78:0/3011666219 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7ff5ac046020 con 0x7ff5b0101800 2024-03-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.980+0000 7ff5a6ffd640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff59c0a2890 0x7ff59c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.980+0000 7ff5b568c640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff59c0a2890 0x7ff59c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.980+0000 7ff5a6ffd640 1 -- 172.21.15.78:0/3011666219 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff5ac0f0be0 con 0x7ff5b0101800 2024-03-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.981+0000 7ff5b568c640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff59c0a2890 0x7ff59c0a4d50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0198f70 tx=0x7ff5a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:56.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:56.985+0000 7ff5a6ffd640 1 -- 172.21.15.78:0/3011666219 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff5ac04d020 con 0x7ff5b0101800 2024-03-16T11:05:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.201+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5b002d1b0 con 0x7ff5b0101800 2024-03-16T11:05:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.202+0000 7ff5a6ffd640 1 -- 172.21.15.78:0/3011666219 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7ff5ac0bdc30 con 0x7ff5b0101800 2024-03-16T11:05:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:57.203 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:05:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:05:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff59c0a2890 msgr2=0x7ff59c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff59c0a2890 0x7ff59c0a4d50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0198f70 tx=0x7ff5a0005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 msgr2=0x7ff5b0197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0197a60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac000c00 tx=0x7ff5ac005dc0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 shutdown_connections 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff59c0a2890 0x7ff59c0a4d50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5b01033b0 0x7ff5b019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5b01029f0 0x7ff5b0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 --2- 172.21.15.78:0/3011666219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5b0101800 0x7ff5b0197a60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 >> 172.21.15.78:0/3011666219 conn(0x7ff5b00fcd90 msgr2=0x7ff5b0105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 shutdown_connections 2024-03-16T11:05:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.204+0000 7ff5b80f1640 1 -- 172.21.15.78:0/3011666219 wait complete. 2024-03-16T11:05:57.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:57 smithi078 ceph-mon[32235]: pgmap v347: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-16T11:05:57.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:57 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3293348003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:57 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/354591086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:57 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/768457151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.216 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-16T11:05:57.216 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-16T11:05:57.216 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 -- 172.21.15.78:0/3588577073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c1022c0 msgr2=0x7f130c108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 --2- 172.21.15.78:0/3588577073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c1022c0 0x7f130c108b50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1304009ca0 tx=0x7f130402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 -- 172.21.15.78:0/3588577073 shutdown_connections 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 --2- 172.21.15.78:0/3588577073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c1022c0 0x7f130c108b50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 --2- 172.21.15.78:0/3588577073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f130c101900 0x7f130c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 --2- 172.21.15.78:0/3588577073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c100710 0x7f130c100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 -- 172.21.15.78:0/3588577073 >> 172.21.15.78:0/3588577073 conn(0x7f130c0fbcc0 msgr2=0x7f130c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 -- 172.21.15.78:0/3588577073 shutdown_connections 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.317+0000 7f1312434640 1 -- 172.21.15.78:0/3588577073 wait complete. 2024-03-16T11:05:57.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 Processor -- start 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 -- start start 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c100710 0x7f130c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c101900 0x7f130c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f130c1022c0 0x7f130c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f130c10c240 con 0x7f130c101900 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f130c10c0c0 con 0x7f130c1022c0 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f1312434640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f130c10c3c0 con 0x7f130c100710 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c101900 0x7f130c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c100710 0x7f130c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c101900 0x7f130c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54778/0 (socket says 172.21.15.78:54778) 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c100710 0x7f130c197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:41080/0 (socket says 172.21.15.78:41080) 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 -- 172.21.15.78:0/427474474 learned_addr learned my addr 172.21.15.78:0/427474474 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f13109d1640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f130c1022c0 0x7f130c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 -- 172.21.15.78:0/427474474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c100710 msgr2=0x7f130c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c100710 0x7f130c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 -- 172.21.15.78:0/427474474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f130c1022c0 msgr2=0x7f130c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f130c1022c0 0x7f130c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f130b7fe640 1 -- 172.21.15.78:0/427474474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c19f680 con 0x7f130c101900 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.318+0000 7f13109d1640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f130c1022c0 0x7f130c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.319+0000 7f130b7fe640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c101900 0x7f130c197fa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f130000e9f0 tx=0x7f130000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.319+0000 7f13097fa640 1 -- 172.21.15.78:0/427474474 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130000cd80 con 0x7f130c101900 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.319+0000 7f13097fa640 1 -- 172.21.15.78:0/427474474 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13000108e0 con 0x7f130c101900 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.319+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130c19f970 con 0x7f130c101900 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.319+0000 7f13097fa640 1 -- 172.21.15.78:0/427474474 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300018920 con 0x7f130c101900 2024-03-16T11:05:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.319+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f130c19fec0 con 0x7f130c101900 2024-03-16T11:05:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.320+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f130c1a00f0 con 0x7f130c101900 2024-03-16T11:05:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.321+0000 7f13097fa640 1 -- 172.21.15.78:0/427474474 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f1300010a90 con 0x7f130c101900 2024-03-16T11:05:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.321+0000 7f13097fa640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f12f40a2890 0x7f12f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.321+0000 7f130bfff640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f12f40a2890 0x7f12f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.321+0000 7f13097fa640 1 -- 172.21.15.78:0/427474474 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f13000cef10 con 0x7f130c101900 2024-03-16T11:05:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.322+0000 7f130bfff640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f12f40a2890 0x7f12f40a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f130c101760 tx=0x7f12fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.326+0000 7f13097fa640 1 -- 172.21.15.78:0/427474474 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f130009b800 con 0x7f130c101900 2024-03-16T11:05:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:57 smithi098 ceph-mon[35848]: pgmap v347: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-16T11:05:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:57 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3293348003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:57 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/354591086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:57 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/768457151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.497+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f130c0fdef0 con 0x7f12f40a2890 2024-03-16T11:05:57.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.505+0000 7f13097fa640 1 -- 172.21.15.78:0/427474474 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258834 (secure 0 0 0) 0x7f130c0fdef0 con 0x7f12f40a2890 2024-03-16T11:05:57.508 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f12f40a2890 msgr2=0x7f12f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f12f40a2890 0x7f12f40a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f130c101760 tx=0x7f12fc005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c101900 msgr2=0x7f130c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c101900 0x7f130c197fa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f130000e9f0 tx=0x7f130000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 shutdown_connections 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f130c1022c0 0x7f130c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f130c101900 0x7f130c197fa0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f12f40a2890 0x7f12f40a4d50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 --2- 172.21.15.78:0/427474474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c100710 0x7f130c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 >> 172.21.15.78:0/427474474 conn(0x7f130c0fbcc0 msgr2=0x7f130c0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:57.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 shutdown_connections 2024-03-16T11:05:57.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.508+0000 7f1312434640 1 -- 172.21.15.78:0/427474474 wait complete. 2024-03-16T11:05:57.519 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-03-16T11:05:55.993515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662415,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":828,"num_read_kb":533,"num_write":353,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383600,"kb_used_data":13648,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558224,"statfs":{"total":1151940427776,"available":1151547621376,"internally_reserved":0,"allocated":13975552,"data_stored":7615042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8510,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":49,"num_write":9,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002827"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008007+0000","last_change":"2024-03-16T11:04:00.295528+0000","last_active":"2024-03-16T11:05:32.008007+0000","last_peered":"2024-03-16T11:05:32.008007+0000","last_clean":"2024-03-16T11:05:32.008007+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:32.008007+0000","last_undegraded":"2024-03-16T11:05:32.008007+0000","last_fullsized":"2024-03-16T11:05:32.008007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:22.852488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994520+0000","last_change":"2024-03-16T11:04:00.289043+0000","last_active":"2024-03-16T11:05:31.994520+0000","last_peered":"2024-03-16T11:05:31.994520+0000","last_clean":"2024-03-16T11:05:31.994520+0000","last_became_active":"2024-03-16T11:04:00.288742+0000","last_became_peered":"2024-03-16T11:04:00.288742+0000","last_unstale":"2024-03-16T11:05:31.994520+0000","last_undegraded":"2024-03-16T11:05:31.994520+0000","last_fullsized":"2024-03-16T11:05:31.994520+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:07:56.710499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007950+0000","last_change":"2024-03-16T11:04:00.293443+0000","last_active":"2024-03-16T11:05:32.007950+0000","last_peered":"2024-03-16T11:05:32.007950+0000","last_clean":"2024-03-16T11:05:32.007950+0000","last_became_active":"2024-03-16T11:04:00.293117+0000","last_became_peered":"2024-03-16T11:04:00.293117+0000","last_unstale":"2024-03-16T11:05:32.007950+0000","last_undegraded":"2024-03-16T11:05:32.007950+0000","last_fullsized":"2024-03-16T11:05:32.007950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:49.728721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998576+0000","last_change":"2024-03-16T11:04:00.292105+0000","last_active":"2024-03-16T11:05:31.998576+0000","last_peered":"2024-03-16T11:05:31.998576+0000","last_clean":"2024-03-16T11:05:31.998576+0000","last_became_active":"2024-03-16T11:04:00.291755+0000","last_became_peered":"2024-03-16T11:04:00.291755+0000","last_unstale":"2024-03-16T11:05:31.998576+0000","last_undegraded":"2024-03-16T11:05:31.998576+0000","last_fullsized":"2024-03-16T11:05:31.998576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:48:43.734854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.997439+0000","last_change":"2024-03-16T11:04:00.282168+0000","last_active":"2024-03-16T11:05:31.997439+0000","last_peered":"2024-03-16T11:05:31.997439+0000","last_clean":"2024-03-16T11:05:31.997439+0000","last_became_active":"2024-03-16T11:04:00.281753+0000","last_became_peered":"2024-03-16T11:04:00.281753+0000","last_unstale":"2024-03-16T11:05:31.997439+0000","last_undegraded":"2024-03-16T11:05:31.997439+0000","last_fullsized":"2024-03-16T11:05:31.997439+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:13.223955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002045+0000","last_change":"2024-03-16T11:05:03.111462+0000","last_active":"2024-03-16T11:05:32.002045+0000","last_peered":"2024-03-16T11:05:32.002045+0000","last_clean":"2024-03-16T11:05:32.002045+0000","last_became_active":"2024-03-16T11:05:03.111087+0000","last_became_peered":"2024-03-16T11:05:03.111087+0000","last_unstale":"2024-03-16T11:05:32.002045+0000","last_undegraded":"2024-03-16T11:05:32.002045+0000","last_fullsized":"2024-03-16T11:05:32.002045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:09.339732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999489+0000","last_change":"2024-03-16T11:04:00.292446+0000","last_active":"2024-03-16T11:05:31.999489+0000","last_peered":"2024-03-16T11:05:31.999489+0000","last_clean":"2024-03-16T11:05:31.999489+0000","last_became_active":"2024-03-16T11:04:00.292079+0000","last_became_peered":"2024-03-16T11:04:00.292079+0000","last_unstale":"2024-03-16T11:05:31.999489+0000","last_undegraded":"2024-03-16T11:05:31.999489+0000","last_fullsized":"2024-03-16T11:05:31.999489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.325945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003952+0000","last_change":"2024-03-16T11:04:00.294644+0000","last_active":"2024-03-16T11:05:32.003952+0000","last_peered":"2024-03-16T11:05:32.003952+0000","last_clean":"2024-03-16T11:05:32.003952+0000","last_became_active":"2024-03-16T11:04:00.294283+0000","last_became_peered":"2024-03-16T11:04:00.294283+0000","last_unstale":"2024-03-16T11:05:32.003952+0000","last_undegraded":"2024-03-16T11:05:32.003952+0000","last_fullsized":"2024-03-16T11:05:32.003952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:43.125823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007001+0000","last_change":"2024-03-16T11:04:00.302512+0000","last_active":"2024-03-16T11:05:32.007001+0000","last_peered":"2024-03-16T11:05:32.007001+0000","last_clean":"2024-03-16T11:05:32.007001+0000","last_became_active":"2024-03-16T11:04:00.301891+0000","last_became_peered":"2024-03-16T11:04:00.301891+0000","last_unstale":"2024-03-16T11:05:32.007001+0000","last_undegraded":"2024-03-16T11:05:32.007001+0000","last_fullsized":"2024-03-16T11:05:32.007001+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:15.112295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003148+0000","last_change":"2024-03-16T11:04:00.293327+0000","last_active":"2024-03-16T11:05:32.003148+0000","last_peered":"2024-03-16T11:05:32.003148+0000","last_clean":"2024-03-16T11:05:32.003148+0000","last_became_active":"2024-03-16T11:04:00.292971+0000","last_became_peered":"2024-03-16T11:04:00.292971+0000","last_unstale":"2024-03-16T11:05:32.003148+0000","last_undegraded":"2024-03-16T11:05:32.003148+0000","last_fullsized":"2024-03-16T11:05:32.003148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:48:48.565861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003814+0000","last_change":"2024-03-16T11:04:00.298956+0000","last_active":"2024-03-16T11:05:32.003814+0000","last_peered":"2024-03-16T11:05:32.003814+0000","last_clean":"2024-03-16T11:05:32.003814+0000","last_became_active":"2024-03-16T11:04:00.298508+0000","last_became_peered":"2024-03-16T11:04:00.298508+0000","last_unstale":"2024-03-16T11:05:32.003814+0000","last_undegraded":"2024-03-16T11:05:32.003814+0000","last_fullsized":"2024-03-16T11:05:32.003814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:25:46.576514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002545+0000","last_change":"2024-03-16T11:04:00.294453+0000","last_active":"2024-03-16T11:05:32.002545+0000","last_peered":"2024-03-16T11:05:32.002545+0000","last_clean":"2024-03-16T11:05:32.002545+0000","last_became_active":"2024-03-16T11:04:00.294084+0000","last_became_peered":"2024-03-16T11:04:00.294084+0000","last_unstale":"2024-03-16T11:05:32.002545+0000","last_undegraded":"2024-03-16T11:05:32.002545+0000","last_fullsized":"2024-03-16T11:05:32.002545+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:12.924835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003591+0000","last_change":"2024-03-16T11:04:00.293149+0000","last_active":"2024-03-16T11:05:32.003591+0000","last_peered":"2024-03-16T11:05:32.003591+0000","last_clean":"2024-03-16T11:05:32.003591+0000","last_became_active":"2024-03-16T11:04:00.292839+0000","last_became_peered":"2024-03-16T11:04:00.292839+0000","last_unstale":"2024-03-16T11:05:32.003591+0000","last_undegraded":"2024-03-16T11:05:32.003591+0000","last_fullsized":"2024-03-16T11:05:32.003591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:34:38.446918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.993502+0000","last_change":"2024-03-16T11:04:00.285748+0000","last_active":"2024-03-16T11:05:31.993502+0000","last_peered":"2024-03-16T11:05:31.993502+0000","last_clean":"2024-03-16T11:05:31.993502+0000","last_became_active":"2024-03-16T11:04:00.285394+0000","last_became_peered":"2024-03-16T11:04:00.285394+0000","last_unstale":"2024-03-16T11:05:31.993502+0000","last_undegraded":"2024-03-16T11:05:31.993502+0000","last_fullsized":"2024-03-16T11:05:31.993502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:02.113271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999076+0000","last_change":"2024-03-16T11:04:00.301783+0000","last_active":"2024-03-16T11:05:31.999076+0000","last_peered":"2024-03-16T11:05:31.999076+0000","last_clean":"2024-03-16T11:05:31.999076+0000","last_became_active":"2024-03-16T11:04:00.301313+0000","last_became_peered":"2024-03-16T11:04:00.301313+0000","last_unstale":"2024-03-16T11:05:31.999076+0000","last_undegraded":"2024-03-16T11:05:31.999076+0000","last_fullsized":"2024-03-16T11:05:31.999076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:19:32.243602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007419+0000","last_change":"2024-03-16T11:04:00.301375+0000","last_active":"2024-03-16T11:05:32.007419+0000","last_peered":"2024-03-16T11:05:32.007419+0000","last_clean":"2024-03-16T11:05:32.007419+0000","last_became_active":"2024-03-16T11:04:00.301013+0000","last_became_peered":"2024-03-16T11:04:00.301013+0000","last_unstale":"2024-03-16T11:05:32.007419+0000","last_undegraded":"2024-03-16T11:05:32.007419+0000","last_fullsized":"2024-03-16T11:05:32.007419+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:21:52.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008074+0000","last_change":"2024-03-16T11:04:00.296287+0000","last_active":"2024-03-16T11:05:32.008074+0000","last_peered":"2024-03-16T11:05:32.008074+0000","last_clean":"2024-03-16T11:05:32.008074+0000","last_became_active":"2024-03-16T11:04:00.295972+0000","last_became_peered":"2024-03-16T11:04:00.295972+0000","last_unstale":"2024-03-16T11:05:32.008074+0000","last_undegraded":"2024-03-16T11:05:32.008074+0000","last_fullsized":"2024-03-16T11:05:32.008074+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:14.586052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006625+0000","last_change":"2024-03-16T11:04:00.295551+0000","last_active":"2024-03-16T11:05:32.006625+0000","last_peered":"2024-03-16T11:05:32.006625+0000","last_clean":"2024-03-16T11:05:32.006625+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:32.006625+0000","last_undegraded":"2024-03-16T11:05:32.006625+0000","last_fullsized":"2024-03-16T11:05:32.006625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:53.495224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.971033+0000","last_change":"2024-03-16T11:04:00.290455+0000","last_active":"2024-03-16T11:05:31.971033+0000","last_peered":"2024-03-16T11:05:31.971033+0000","last_clean":"2024-03-16T11:05:31.971033+0000","last_became_active":"2024-03-16T11:04:00.290057+0000","last_became_peered":"2024-03-16T11:04:00.290057+0000","last_unstale":"2024-03-16T11:05:31.971033+0000","last_undegraded":"2024-03-16T11:05:31.971033+0000","last_fullsized":"2024-03-16T11:05:31.971033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:57:43.677050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102676+0000","last_change":"2024-03-16T11:03:59.285059+0000","last_active":"2024-03-16T11:05:03.102676+0000","last_peered":"2024-03-16T11:05:03.102676+0000","last_clean":"2024-03-16T11:05:03.102676+0000","last_became_active":"2024-03-16T11:03:59.284745+0000","last_became_peered":"2024-03-16T11:03:59.284745+0000","last_unstale":"2024-03-16T11:05:03.102676+0000","last_undegraded":"2024-03-16T11:05:03.102676+0000","last_fullsized":"2024-03-16T11:05:03.102676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:12.516958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990186+0000","last_change":"2024-03-16T11:04:00.292933+0000","last_active":"2024-03-16T11:05:31.990186+0000","last_peered":"2024-03-16T11:05:31.990186+0000","last_clean":"2024-03-16T11:05:31.990186+0000","last_became_active":"2024-03-16T11:04:00.292637+0000","last_became_peered":"2024-03-16T11:04:00.292637+0000","last_unstale":"2024-03-16T11:05:31.990186+0000","last_undegraded":"2024-03-16T11:05:31.990186+0000","last_fullsized":"2024-03-16T11:05:31.990186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:22.231735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107896+0000","last_change":"2024-03-16T11:03:59.285267+0000","last_active":"2024-03-16T11:05:03.107896+0000","last_peered":"2024-03-16T11:05:03.107896+0000","last_clean":"2024-03-16T11:05:03.107896+0000","last_became_active":"2024-03-16T11:03:59.284890+0000","last_became_peered":"2024-03-16T11:03:59.284890+0000","last_unstale":"2024-03-16T11:05:03.107896+0000","last_undegraded":"2024-03-16T11:05:03.107896+0000","last_fullsized":"2024-03-16T11:05:03.107896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:53.625612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008052+0000","last_change":"2024-03-16T11:04:00.292660+0000","last_active":"2024-03-16T11:05:32.008052+0000","last_peered":"2024-03-16T11:05:32.008052+0000","last_clean":"2024-03-16T11:05:32.008052+0000","last_became_active":"2024-03-16T11:04:00.292366+0000","last_became_peered":"2024-03-16T11:04:00.292366+0000","last_unstale":"2024-03-16T11:05:32.008052+0000","last_undegraded":"2024-03-16T11:05:32.008052+0000","last_fullsized":"2024-03-16T11:05:32.008052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:36:44.331456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109296+0000","last_change":"2024-03-16T11:03:59.286370+0000","last_active":"2024-03-16T11:05:03.109296+0000","last_peered":"2024-03-16T11:05:03.109296+0000","last_clean":"2024-03-16T11:05:03.109296+0000","last_became_active":"2024-03-16T11:03:59.286065+0000","last_became_peered":"2024-03-16T11:03:59.286065+0000","last_unstale":"2024-03-16T11:05:03.109296+0000","last_undegraded":"2024-03-16T11:05:03.109296+0000","last_fullsized":"2024-03-16T11:05:03.109296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:20.126848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007484+0000","last_change":"2024-03-16T11:04:00.290864+0000","last_active":"2024-03-16T11:05:32.007484+0000","last_peered":"2024-03-16T11:05:32.007484+0000","last_clean":"2024-03-16T11:05:32.007484+0000","last_became_active":"2024-03-16T11:04:00.290537+0000","last_became_peered":"2024-03-16T11:04:00.290537+0000","last_unstale":"2024-03-16T11:05:32.007484+0000","last_undegraded":"2024-03-16T11:05:32.007484+0000","last_fullsized":"2024-03-16T11:05:32.007484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:52.991767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109155+0000","last_change":"2024-03-16T11:03:59.281663+0000","last_active":"2024-03-16T11:05:03.109155+0000","last_peered":"2024-03-16T11:05:03.109155+0000","last_clean":"2024-03-16T11:05:03.109155+0000","last_became_active":"2024-03-16T11:03:59.281328+0000","last_became_peered":"2024-03-16T11:03:59.281328+0000","last_unstale":"2024-03-16T11:05:03.109155+0000","last_undegraded":"2024-03-16T11:05:03.109155+0000","last_fullsized":"2024-03-16T11:05:03.109155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:44.511109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003970+0000","last_change":"2024-03-16T11:05:03.106586+0000","last_active":"2024-03-16T11:05:32.003970+0000","last_peered":"2024-03-16T11:05:32.003970+0000","last_clean":"2024-03-16T11:05:32.003970+0000","last_became_active":"2024-03-16T11:05:03.106233+0000","last_became_peered":"2024-03-16T11:05:03.106233+0000","last_unstale":"2024-03-16T11:05:32.003970+0000","last_undegraded":"2024-03-16T11:05:32.003970+0000","last_fullsized":"2024-03-16T11:05:32.003970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:19:30.480318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":142,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:54.588491+0000","last_change":"2024-03-16T11:03:59.286300+0000","last_active":"2024-03-16T11:05:54.588491+0000","last_peered":"2024-03-16T11:05:54.588491+0000","last_clean":"2024-03-16T11:05:54.588491+0000","last_became_active":"2024-03-16T11:03:59.285764+0000","last_became_peered":"2024-03-16T11:03:59.285764+0000","last_unstale":"2024-03-16T11:05:54.588491+0000","last_undegraded":"2024-03-16T11:05:54.588491+0000","last_fullsized":"2024-03-16T11:05:54.588491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:49.084245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.984380+0000","last_change":"2024-03-16T11:04:00.289152+0000","last_active":"2024-03-16T11:05:31.984380+0000","last_peered":"2024-03-16T11:05:31.984380+0000","last_clean":"2024-03-16T11:05:31.984380+0000","last_became_active":"2024-03-16T11:04:00.288837+0000","last_became_peered":"2024-03-16T11:04:00.288837+0000","last_unstale":"2024-03-16T11:05:31.984380+0000","last_undegraded":"2024-03-16T11:05:31.984380+0000","last_fullsized":"2024-03-16T11:05:31.984380+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:45:01.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106073+0000","last_change":"2024-03-16T11:03:59.281505+0000","last_active":"2024-03-16T11:05:03.106073+0000","last_peered":"2024-03-16T11:05:03.106073+0000","last_clean":"2024-03-16T11:05:03.106073+0000","last_became_active":"2024-03-16T11:03:59.281175+0000","last_became_peered":"2024-03-16T11:03:59.281175+0000","last_unstale":"2024-03-16T11:05:03.106073+0000","last_undegraded":"2024-03-16T11:05:03.106073+0000","last_fullsized":"2024-03-16T11:05:03.106073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:12.463914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.988807+0000","last_change":"2024-03-16T11:04:00.297769+0000","last_active":"2024-03-16T11:05:31.988807+0000","last_peered":"2024-03-16T11:05:31.988807+0000","last_clean":"2024-03-16T11:05:31.988807+0000","last_became_active":"2024-03-16T11:04:00.297061+0000","last_became_peered":"2024-03-16T11:04:00.297061+0000","last_unstale":"2024-03-16T11:05:31.988807+0000","last_undegraded":"2024-03-16T11:05:31.988807+0000","last_fullsized":"2024-03-16T11:05:31.988807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:42.594315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103214+0000","last_change":"2024-03-16T11:03:59.290260+0000","last_active":"2024-03-16T11:05:03.103214+0000","last_peered":"2024-03-16T11:05:03.103214+0000","last_clean":"2024-03-16T11:05:03.103214+0000","last_became_active":"2024-03-16T11:03:59.289843+0000","last_became_peered":"2024-03-16T11:03:59.289843+0000","last_unstale":"2024-03-16T11:05:03.103214+0000","last_undegraded":"2024-03-16T11:05:03.103214+0000","last_fullsized":"2024-03-16T11:05:03.103214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:22.483488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.997150+0000","last_change":"2024-03-16T11:04:00.287873+0000","last_active":"2024-03-16T11:05:31.997150+0000","last_peered":"2024-03-16T11:05:31.997150+0000","last_clean":"2024-03-16T11:05:31.997150+0000","last_became_active":"2024-03-16T11:04:00.287002+0000","last_became_peered":"2024-03-16T11:04:00.287002+0000","last_unstale":"2024-03-16T11:05:31.997150+0000","last_undegraded":"2024-03-16T11:05:31.997150+0000","last_fullsized":"2024-03-16T11:05:31.997150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:57.851726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102815+0000","last_change":"2024-03-16T11:03:59.284745+0000","last_active":"2024-03-16T11:05:03.102815+0000","last_peered":"2024-03-16T11:05:03.102815+0000","last_clean":"2024-03-16T11:05:03.102815+0000","last_became_active":"2024-03-16T11:03:59.284325+0000","last_became_peered":"2024-03-16T11:03:59.284325+0000","last_unstale":"2024-03-16T11:05:03.102815+0000","last_undegraded":"2024-03-16T11:05:03.102815+0000","last_fullsized":"2024-03-16T11:05:03.102815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.537787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002828+0000","last_change":"2024-03-16T11:04:00.290698+0000","last_active":"2024-03-16T11:05:32.002828+0000","last_peered":"2024-03-16T11:05:32.002828+0000","last_clean":"2024-03-16T11:05:32.002828+0000","last_became_active":"2024-03-16T11:04:00.290348+0000","last_became_peered":"2024-03-16T11:04:00.290348+0000","last_unstale":"2024-03-16T11:05:32.002828+0000","last_undegraded":"2024-03-16T11:05:32.002828+0000","last_fullsized":"2024-03-16T11:05:32.002828+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:44.114305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":135,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.586690+0000","last_change":"2024-03-16T11:03:59.280705+0000","last_active":"2024-03-16T11:05:53.586690+0000","last_peered":"2024-03-16T11:05:53.586690+0000","last_clean":"2024-03-16T11:05:53.586690+0000","last_became_active":"2024-03-16T11:03:59.280372+0000","last_became_peered":"2024-03-16T11:03:59.280372+0000","last_unstale":"2024-03-16T11:05:53.586690+0000","last_undegraded":"2024-03-16T11:05:53.586690+0000","last_fullsized":"2024-03-16T11:05:53.586690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:54.187322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006386+0000","last_change":"2024-03-16T11:04:00.293867+0000","last_active":"2024-03-16T11:05:32.006386+0000","last_peered":"2024-03-16T11:05:32.006386+0000","last_clean":"2024-03-16T11:05:32.006386+0000","last_became_active":"2024-03-16T11:04:00.293330+0000","last_became_peered":"2024-03-16T11:04:00.293330+0000","last_unstale":"2024-03-16T11:05:32.006386+0000","last_undegraded":"2024-03-16T11:05:32.006386+0000","last_fullsized":"2024-03-16T11:05:32.006386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:54.798346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105547+0000","last_change":"2024-03-16T11:03:59.284202+0000","last_active":"2024-03-16T11:05:03.105547+0000","last_peered":"2024-03-16T11:05:03.105547+0000","last_clean":"2024-03-16T11:05:03.105547+0000","last_became_active":"2024-03-16T11:03:59.283656+0000","last_became_peered":"2024-03-16T11:03:59.283656+0000","last_unstale":"2024-03-16T11:05:03.105547+0000","last_undegraded":"2024-03-16T11:05:03.105547+0000","last_fullsized":"2024-03-16T11:05:03.105547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:44.760307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.996327+0000","last_change":"2024-03-16T11:05:03.111443+0000","last_active":"2024-03-16T11:05:31.996327+0000","last_peered":"2024-03-16T11:05:31.996327+0000","last_clean":"2024-03-16T11:05:31.996327+0000","last_became_active":"2024-03-16T11:05:03.111027+0000","last_became_peered":"2024-03-16T11:05:03.111027+0000","last_unstale":"2024-03-16T11:05:31.996327+0000","last_undegraded":"2024-03-16T11:05:31.996327+0000","last_fullsized":"2024-03-16T11:05:31.996327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:31.844283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103330+0000","last_change":"2024-03-16T11:03:59.288268+0000","last_active":"2024-03-16T11:05:03.103330+0000","last_peered":"2024-03-16T11:05:03.103330+0000","last_clean":"2024-03-16T11:05:03.103330+0000","last_became_active":"2024-03-16T11:03:59.287861+0000","last_became_peered":"2024-03-16T11:03:59.287861+0000","last_unstale":"2024-03-16T11:05:03.103330+0000","last_undegraded":"2024-03-16T11:05:03.103330+0000","last_fullsized":"2024-03-16T11:05:03.103330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:34:34.248408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007936+0000","last_change":"2024-03-16T11:04:00.306490+0000","last_active":"2024-03-16T11:05:32.007936+0000","last_peered":"2024-03-16T11:05:32.007936+0000","last_clean":"2024-03-16T11:05:32.007936+0000","last_became_active":"2024-03-16T11:04:00.305874+0000","last_became_peered":"2024-03-16T11:04:00.305874+0000","last_unstale":"2024-03-16T11:05:32.007936+0000","last_undegraded":"2024-03-16T11:05:32.007936+0000","last_fullsized":"2024-03-16T11:05:32.007936+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":286,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:54.589113+0000","last_change":"2024-03-16T11:03:59.280714+0000","last_active":"2024-03-16T11:05:54.589113+0000","last_peered":"2024-03-16T11:05:54.589113+0000","last_clean":"2024-03-16T11:05:54.589113+0000","last_became_active":"2024-03-16T11:03:59.280398+0000","last_became_peered":"2024-03-16T11:03:59.280398+0000","last_unstale":"2024-03-16T11:05:54.589113+0000","last_undegraded":"2024-03-16T11:05:54.589113+0000","last_fullsized":"2024-03-16T11:05:54.589113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:34.256125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66294,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":234,"num_read_kb":179,"num_write":34,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999739+0000","last_change":"2024-03-16T11:04:00.285669+0000","last_active":"2024-03-16T11:05:31.999739+0000","last_peered":"2024-03-16T11:05:31.999739+0000","last_clean":"2024-03-16T11:05:31.999739+0000","last_became_active":"2024-03-16T11:04:00.285337+0000","last_became_peered":"2024-03-16T11:04:00.285337+0000","last_unstale":"2024-03-16T11:05:31.999739+0000","last_undegraded":"2024-03-16T11:05:31.999739+0000","last_fullsized":"2024-03-16T11:05:31.999739+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:36.925350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548506+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548506+0000","last_peered":"2024-03-16T11:05:03.548506+0000","last_clean":"2024-03-16T11:05:03.548506+0000","last_became_active":"2024-03-16T11:03:59.289357+0000","last_became_peered":"2024-03-16T11:03:59.289357+0000","last_unstale":"2024-03-16T11:05:03.548506+0000","last_undegraded":"2024-03-16T11:05:03.548506+0000","last_fullsized":"2024-03-16T11:05:03.548506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:26:35.860091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998250+0000","last_change":"2024-03-16T11:04:00.298488+0000","last_active":"2024-03-16T11:05:31.998250+0000","last_peered":"2024-03-16T11:05:31.998250+0000","last_clean":"2024-03-16T11:05:31.998250+0000","last_became_active":"2024-03-16T11:04:00.298202+0000","last_became_peered":"2024-03-16T11:04:00.298202+0000","last_unstale":"2024-03-16T11:05:31.998250+0000","last_undegraded":"2024-03-16T11:05:31.998250+0000","last_fullsized":"2024-03-16T11:05:31.998250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:48.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102961+0000","last_change":"2024-03-16T11:03:59.287066+0000","last_active":"2024-03-16T11:05:03.102961+0000","last_peered":"2024-03-16T11:05:03.102961+0000","last_clean":"2024-03-16T11:05:03.102961+0000","last_became_active":"2024-03-16T11:03:59.286731+0000","last_became_peered":"2024-03-16T11:03:59.286731+0000","last_unstale":"2024-03-16T11:05:03.102961+0000","last_undegraded":"2024-03-16T11:05:03.102961+0000","last_fullsized":"2024-03-16T11:05:03.102961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:51.443833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999855+0000","last_change":"2024-03-16T11:04:00.294450+0000","last_active":"2024-03-16T11:05:31.999855+0000","last_peered":"2024-03-16T11:05:31.999855+0000","last_clean":"2024-03-16T11:05:31.999855+0000","last_became_active":"2024-03-16T11:04:00.293769+0000","last_became_peered":"2024-03-16T11:04:00.293769+0000","last_unstale":"2024-03-16T11:05:31.999855+0000","last_undegraded":"2024-03-16T11:05:31.999855+0000","last_fullsized":"2024-03-16T11:05:31.999855+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:02.755907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":141,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.597588+0000","last_change":"2024-03-16T11:03:59.286197+0000","last_active":"2024-03-16T11:05:53.597588+0000","last_peered":"2024-03-16T11:05:53.597588+0000","last_clean":"2024-03-16T11:05:53.597588+0000","last_became_active":"2024-03-16T11:03:59.285757+0000","last_became_peered":"2024-03-16T11:03:59.285757+0000","last_unstale":"2024-03-16T11:05:53.597588+0000","last_undegraded":"2024-03-16T11:05:53.597588+0000","last_fullsized":"2024-03-16T11:05:53.597588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:47.868106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002798+0000","last_change":"2024-03-16T11:04:00.296709+0000","last_active":"2024-03-16T11:05:32.002798+0000","last_peered":"2024-03-16T11:05:32.002798+0000","last_clean":"2024-03-16T11:05:32.002798+0000","last_became_active":"2024-03-16T11:04:00.296372+0000","last_became_peered":"2024-03-16T11:04:00.296372+0000","last_unstale":"2024-03-16T11:05:32.002798+0000","last_undegraded":"2024-03-16T11:05:32.002798+0000","last_fullsized":"2024-03-16T11:05:32.002798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:37.948866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108997+0000","last_change":"2024-03-16T11:03:59.283831+0000","last_active":"2024-03-16T11:05:03.108997+0000","last_peered":"2024-03-16T11:05:03.108997+0000","last_clean":"2024-03-16T11:05:03.108997+0000","last_became_active":"2024-03-16T11:03:59.283404+0000","last_became_peered":"2024-03-16T11:03:59.283404+0000","last_unstale":"2024-03-16T11:05:03.108997+0000","last_undegraded":"2024-03-16T11:05:03.108997+0000","last_fullsized":"2024-03-16T11:05:03.108997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:26:34.012597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.984955+0000","last_change":"2024-03-16T11:04:00.299703+0000","last_active":"2024-03-16T11:05:31.984955+0000","last_peered":"2024-03-16T11:05:31.984955+0000","last_clean":"2024-03-16T11:05:31.984955+0000","last_became_active":"2024-03-16T11:04:00.299321+0000","last_became_peered":"2024-03-16T11:04:00.299321+0000","last_unstale":"2024-03-16T11:05:31.984955+0000","last_undegraded":"2024-03-16T11:05:31.984955+0000","last_fullsized":"2024-03-16T11:05:31.984955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.529281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109510+0000","last_change":"2024-03-16T11:03:59.277837+0000","last_active":"2024-03-16T11:05:03.109510+0000","last_peered":"2024-03-16T11:05:03.109510+0000","last_clean":"2024-03-16T11:05:03.109510+0000","last_became_active":"2024-03-16T11:03:59.277515+0000","last_became_peered":"2024-03-16T11:03:59.277515+0000","last_unstale":"2024-03-16T11:05:03.109510+0000","last_undegraded":"2024-03-16T11:05:03.109510+0000","last_fullsized":"2024-03-16T11:05:03.109510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:05.116990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994100+0000","last_change":"2024-03-16T11:04:00.292998+0000","last_active":"2024-03-16T11:05:31.994100+0000","last_peered":"2024-03-16T11:05:31.994100+0000","last_clean":"2024-03-16T11:05:31.994100+0000","last_became_active":"2024-03-16T11:04:00.292608+0000","last_became_peered":"2024-03-16T11:04:00.292608+0000","last_unstale":"2024-03-16T11:05:31.994100+0000","last_undegraded":"2024-03-16T11:05:31.994100+0000","last_fullsized":"2024-03-16T11:05:31.994100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:41.996826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108505+0000","last_change":"2024-03-16T11:05:03.108505+0000","last_active":"2024-03-16T11:05:03.108505+0000","last_peered":"2024-03-16T11:05:03.108505+0000","last_clean":"2024-03-16T11:05:03.108505+0000","last_became_active":"2024-03-16T11:05:03.108132+0000","last_became_peered":"2024-03-16T11:05:03.108132+0000","last_unstale":"2024-03-16T11:05:03.108505+0000","last_undegraded":"2024-03-16T11:05:03.108505+0000","last_fullsized":"2024-03-16T11:05:03.108505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:50.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002870+0000","last_change":"2024-03-16T11:04:00.288897+0000","last_active":"2024-03-16T11:05:32.002870+0000","last_peered":"2024-03-16T11:05:32.002870+0000","last_clean":"2024-03-16T11:05:32.002870+0000","last_became_active":"2024-03-16T11:04:00.288493+0000","last_became_peered":"2024-03-16T11:04:00.288493+0000","last_unstale":"2024-03-16T11:05:32.002870+0000","last_undegraded":"2024-03-16T11:05:32.002870+0000","last_fullsized":"2024-03-16T11:05:32.002870+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:31.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105515+0000","last_change":"2024-03-16T11:03:59.291595+0000","last_active":"2024-03-16T11:05:03.105515+0000","last_peered":"2024-03-16T11:05:03.105515+0000","last_clean":"2024-03-16T11:05:03.105515+0000","last_became_active":"2024-03-16T11:03:59.291159+0000","last_became_peered":"2024-03-16T11:03:59.291159+0000","last_unstale":"2024-03-16T11:05:03.105515+0000","last_undegraded":"2024-03-16T11:05:03.105515+0000","last_fullsized":"2024-03-16T11:05:03.105515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:03.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000305+0000","last_change":"2024-03-16T11:05:03.108818+0000","last_active":"2024-03-16T11:05:32.000305+0000","last_peered":"2024-03-16T11:05:32.000305+0000","last_clean":"2024-03-16T11:05:32.000305+0000","last_became_active":"2024-03-16T11:05:03.108479+0000","last_became_peered":"2024-03-16T11:05:03.108479+0000","last_unstale":"2024-03-16T11:05:32.000305+0000","last_undegraded":"2024-03-16T11:05:32.000305+0000","last_fullsized":"2024-03-16T11:05:32.000305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:10:11.329743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.977502+0000","last_change":"2024-03-16T11:04:00.298400+0000","last_active":"2024-03-16T11:05:31.977502+0000","last_peered":"2024-03-16T11:05:31.977502+0000","last_clean":"2024-03-16T11:05:31.977502+0000","last_became_active":"2024-03-16T11:04:00.298026+0000","last_became_peered":"2024-03-16T11:04:00.298026+0000","last_unstale":"2024-03-16T11:05:31.977502+0000","last_undegraded":"2024-03-16T11:05:31.977502+0000","last_fullsized":"2024-03-16T11:05:31.977502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:08.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":136,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.225688+0000","last_change":"2024-03-16T11:03:59.287528+0000","last_active":"2024-03-16T11:05:53.225688+0000","last_peered":"2024-03-16T11:05:53.225688+0000","last_clean":"2024-03-16T11:05:53.225688+0000","last_became_active":"2024-03-16T11:03:59.287218+0000","last_became_peered":"2024-03-16T11:03:59.287218+0000","last_unstale":"2024-03-16T11:05:53.225688+0000","last_undegraded":"2024-03-16T11:05:53.225688+0000","last_fullsized":"2024-03-16T11:05:53.225688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:19.803233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006796+0000","last_change":"2024-03-16T11:05:03.106099+0000","last_active":"2024-03-16T11:05:32.006796+0000","last_peered":"2024-03-16T11:05:32.006796+0000","last_clean":"2024-03-16T11:05:32.006796+0000","last_became_active":"2024-03-16T11:05:03.105412+0000","last_became_peered":"2024-03-16T11:05:03.105412+0000","last_unstale":"2024-03-16T11:05:32.006796+0000","last_undegraded":"2024-03-16T11:05:32.006796+0000","last_fullsized":"2024-03-16T11:05:32.006796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:29:33.213676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.541779+0000","last_change":"2024-03-16T11:03:59.281251+0000","last_active":"2024-03-16T11:05:03.541779+0000","last_peered":"2024-03-16T11:05:03.541779+0000","last_clean":"2024-03-16T11:05:03.541779+0000","last_became_active":"2024-03-16T11:03:59.280830+0000","last_became_peered":"2024-03-16T11:03:59.280830+0000","last_unstale":"2024-03-16T11:05:03.541779+0000","last_undegraded":"2024-03-16T11:05:03.541779+0000","last_fullsized":"2024-03-16T11:05:03.541779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:03:04.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105473+0000","last_change":"2024-03-16T11:02:48.497749+0000","last_active":"2024-03-16T11:05:03.105473+0000","last_peered":"2024-03-16T11:05:03.105473+0000","last_clean":"2024-03-16T11:05:03.105473+0000","last_became_active":"2024-03-16T11:02:48.497157+0000","last_became_peered":"2024-03-16T11:02:48.497157+0000","last_unstale":"2024-03-16T11:05:03.105473+0000","last_undegraded":"2024-03-16T11:05:03.105473+0000","last_fullsized":"2024-03-16T11:05:03.105473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_clean_scrub_stamp":"2024-03-16T10:59:26.309639+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:04.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999743+0000","last_change":"2024-03-16T11:04:00.300821+0000","last_active":"2024-03-16T11:05:31.999743+0000","last_peered":"2024-03-16T11:05:31.999743+0000","last_clean":"2024-03-16T11:05:31.999743+0000","last_became_active":"2024-03-16T11:04:00.300510+0000","last_became_peered":"2024-03-16T11:04:00.300510+0000","last_unstale":"2024-03-16T11:05:31.999743+0000","last_undegraded":"2024-03-16T11:05:31.999743+0000","last_fullsized":"2024-03-16T11:05:31.999743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:57.035167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105571+0000","last_change":"2024-03-16T11:03:59.283609+0000","last_active":"2024-03-16T11:05:03.105571+0000","last_peered":"2024-03-16T11:05:03.105571+0000","last_clean":"2024-03-16T11:05:03.105571+0000","last_became_active":"2024-03-16T11:03:59.283227+0000","last_became_peered":"2024-03-16T11:03:59.283227+0000","last_unstale":"2024-03-16T11:05:03.105571+0000","last_undegraded":"2024-03-16T11:05:03.105571+0000","last_fullsized":"2024-03-16T11:05:03.105571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:18.904202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.112457+0000","last_change":"2024-03-16T11:05:03.112071+0000","last_active":"2024-03-16T11:05:03.112457+0000","last_peered":"2024-03-16T11:05:03.112457+0000","last_clean":"2024-03-16T11:05:03.112457+0000","last_became_active":"2024-03-16T11:05:03.108434+0000","last_became_peered":"2024-03-16T11:05:03.108434+0000","last_unstale":"2024-03-16T11:05:03.112457+0000","last_undegraded":"2024-03-16T11:05:03.112457+0000","last_fullsized":"2024-03-16T11:05:03.112457+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:24.758886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000550+0000","last_change":"2024-03-16T11:04:00.285134+0000","last_active":"2024-03-16T11:05:32.000550+0000","last_peered":"2024-03-16T11:05:32.000550+0000","last_clean":"2024-03-16T11:05:32.000550+0000","last_became_active":"2024-03-16T11:04:00.284705+0000","last_became_peered":"2024-03-16T11:04:00.284705+0000","last_unstale":"2024-03-16T11:05:32.000550+0000","last_undegraded":"2024-03-16T11:05:32.000550+0000","last_fullsized":"2024-03-16T11:05:32.000550+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:38.637970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102785+0000","last_change":"2024-03-16T11:03:59.289999+0000","last_active":"2024-03-16T11:05:03.102785+0000","last_peered":"2024-03-16T11:05:03.102785+0000","last_clean":"2024-03-16T11:05:03.102785+0000","last_became_active":"2024-03-16T11:03:59.289694+0000","last_became_peered":"2024-03-16T11:03:59.289694+0000","last_unstale":"2024-03-16T11:05:03.102785+0000","last_undegraded":"2024-03-16T11:05:03.102785+0000","last_fullsized":"2024-03-16T11:05:03.102785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:16.037387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000119+0000","last_change":"2024-03-16T11:04:00.298148+0000","last_active":"2024-03-16T11:05:32.000119+0000","last_peered":"2024-03-16T11:05:32.000119+0000","last_clean":"2024-03-16T11:05:32.000119+0000","last_became_active":"2024-03-16T11:04:00.297822+0000","last_became_peered":"2024-03-16T11:04:00.297822+0000","last_unstale":"2024-03-16T11:05:32.000119+0000","last_undegraded":"2024-03-16T11:05:32.000119+0000","last_fullsized":"2024-03-16T11:05:32.000119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:09:28.645317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542005+0000","last_change":"2024-03-16T11:03:59.292207+0000","last_active":"2024-03-16T11:05:03.542005+0000","last_peered":"2024-03-16T11:05:03.542005+0000","last_clean":"2024-03-16T11:05:03.542005+0000","last_became_active":"2024-03-16T11:03:59.291859+0000","last_became_peered":"2024-03-16T11:03:59.291859+0000","last_unstale":"2024-03-16T11:05:03.542005+0000","last_undegraded":"2024-03-16T11:05:03.542005+0000","last_fullsized":"2024-03-16T11:05:03.542005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:55.365720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998794+0000","last_change":"2024-03-16T11:04:00.291261+0000","last_active":"2024-03-16T11:05:31.998794+0000","last_peered":"2024-03-16T11:05:31.998794+0000","last_clean":"2024-03-16T11:05:31.998794+0000","last_became_active":"2024-03-16T11:04:00.290832+0000","last_became_peered":"2024-03-16T11:04:00.290832+0000","last_unstale":"2024-03-16T11:05:31.998794+0000","last_undegraded":"2024-03-16T11:05:31.998794+0000","last_fullsized":"2024-03-16T11:05:31.998794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:10.418808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.100986+0000","last_change":"2024-03-16T11:03:59.288014+0000","last_active":"2024-03-16T11:05:03.100986+0000","last_peered":"2024-03-16T11:05:03.100986+0000","last_clean":"2024-03-16T11:05:03.100986+0000","last_became_active":"2024-03-16T11:03:59.287679+0000","last_became_peered":"2024-03-16T11:03:59.287679+0000","last_unstale":"2024-03-16T11:05:03.100986+0000","last_undegraded":"2024-03-16T11:05:03.100986+0000","last_fullsized":"2024-03-16T11:05:03.100986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:16:39.156224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.960063+0000","last_change":"2024-03-16T11:04:00.288047+0000","last_active":"2024-03-16T11:05:31.960063+0000","last_peered":"2024-03-16T11:05:31.960063+0000","last_clean":"2024-03-16T11:05:31.960063+0000","last_became_active":"2024-03-16T11:04:00.287619+0000","last_became_peered":"2024-03-16T11:04:00.287619+0000","last_unstale":"2024-03-16T11:05:31.960063+0000","last_undegraded":"2024-03-16T11:05:31.960063+0000","last_fullsized":"2024-03-16T11:05:31.960063+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:43.608349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106759+0000","last_change":"2024-03-16T11:03:59.288439+0000","last_active":"2024-03-16T11:05:03.106759+0000","last_peered":"2024-03-16T11:05:03.106759+0000","last_clean":"2024-03-16T11:05:03.106759+0000","last_became_active":"2024-03-16T11:03:59.288122+0000","last_became_peered":"2024-03-16T11:03:59.288122+0000","last_unstale":"2024-03-16T11:05:03.106759+0000","last_undegraded":"2024-03-16T11:05:03.106759+0000","last_fullsized":"2024-03-16T11:05:03.106759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:50:25.499608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998940+0000","last_change":"2024-03-16T11:04:00.290522+0000","last_active":"2024-03-16T11:05:31.998940+0000","last_peered":"2024-03-16T11:05:31.998940+0000","last_clean":"2024-03-16T11:05:31.998940+0000","last_became_active":"2024-03-16T11:04:00.290175+0000","last_became_peered":"2024-03-16T11:04:00.290175+0000","last_unstale":"2024-03-16T11:05:31.998940+0000","last_undegraded":"2024-03-16T11:05:31.998940+0000","last_fullsized":"2024-03-16T11:05:31.998940+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:34.113278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548452+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548452+0000","last_peered":"2024-03-16T11:05:03.548452+0000","last_clean":"2024-03-16T11:05:03.548452+0000","last_became_active":"2024-03-16T11:03:59.289367+0000","last_became_peered":"2024-03-16T11:03:59.289367+0000","last_unstale":"2024-03-16T11:05:03.548452+0000","last_undegraded":"2024-03-16T11:05:03.548452+0000","last_fullsized":"2024-03-16T11:05:03.548452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:42:48.627116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994427+0000","last_change":"2024-03-16T11:04:00.287685+0000","last_active":"2024-03-16T11:05:31.994427+0000","last_peered":"2024-03-16T11:05:31.994427+0000","last_clean":"2024-03-16T11:05:31.994427+0000","last_became_active":"2024-03-16T11:04:00.287367+0000","last_became_peered":"2024-03-16T11:04:00.287367+0000","last_unstale":"2024-03-16T11:05:31.994427+0000","last_undegraded":"2024-03-16T11:05:31.994427+0000","last_fullsized":"2024-03-16T11:05:31.994427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:25:52.741651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108923+0000","last_change":"2024-03-16T11:03:59.286293+0000","last_active":"2024-03-16T11:05:03.108923+0000","last_peered":"2024-03-16T11:05:03.108923+0000","last_clean":"2024-03-16T11:05:03.108923+0000","last_became_active":"2024-03-16T11:03:59.285817+0000","last_became_peered":"2024-03-16T11:03:59.285817+0000","last_unstale":"2024-03-16T11:05:03.108923+0000","last_undegraded":"2024-03-16T11:05:03.108923+0000","last_fullsized":"2024-03-16T11:05:03.108923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:38:12.093762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008486+0000","last_change":"2024-03-16T11:04:00.299391+0000","last_active":"2024-03-16T11:05:32.008486+0000","last_peered":"2024-03-16T11:05:32.008486+0000","last_clean":"2024-03-16T11:05:32.008486+0000","last_became_active":"2024-03-16T11:04:00.298931+0000","last_became_peered":"2024-03-16T11:04:00.298931+0000","last_unstale":"2024-03-16T11:05:32.008486+0000","last_undegraded":"2024-03-16T11:05:32.008486+0000","last_fullsized":"2024-03-16T11:05:32.008486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:12:11.900154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548534+0000","last_change":"2024-03-16T11:03:59.292319+0000","last_active":"2024-03-16T11:05:03.548534+0000","last_peered":"2024-03-16T11:05:03.548534+0000","last_clean":"2024-03-16T11:05:03.548534+0000","last_became_active":"2024-03-16T11:03:59.291953+0000","last_became_peered":"2024-03-16T11:03:59.291953+0000","last_unstale":"2024-03-16T11:05:03.548534+0000","last_undegraded":"2024-03-16T11:05:03.548534+0000","last_fullsized":"2024-03-16T11:05:03.548534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:41:04.615726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008131+0000","last_change":"2024-03-16T11:04:00.296544+0000","last_active":"2024-03-16T11:05:32.008131+0000","last_peered":"2024-03-16T11:05:32.008131+0000","last_clean":"2024-03-16T11:05:32.008131+0000","last_became_active":"2024-03-16T11:04:00.296078+0000","last_became_peered":"2024-03-16T11:04:00.296078+0000","last_unstale":"2024-03-16T11:05:32.008131+0000","last_undegraded":"2024-03-16T11:05:32.008131+0000","last_fullsized":"2024-03-16T11:05:32.008131+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:56:22.846734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103250+0000","last_change":"2024-03-16T11:03:59.287172+0000","last_active":"2024-03-16T11:05:03.103250+0000","last_peered":"2024-03-16T11:05:03.103250+0000","last_clean":"2024-03-16T11:05:03.103250+0000","last_became_active":"2024-03-16T11:03:59.286785+0000","last_became_peered":"2024-03-16T11:03:59.286785+0000","last_unstale":"2024-03-16T11:05:03.103250+0000","last_undegraded":"2024-03-16T11:05:03.103250+0000","last_fullsized":"2024-03-16T11:05:03.103250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:17:55.584618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990540+0000","last_change":"2024-03-16T11:05:03.111534+0000","last_active":"2024-03-16T11:05:31.990540+0000","last_peered":"2024-03-16T11:05:31.990540+0000","last_clean":"2024-03-16T11:05:31.990540+0000","last_became_active":"2024-03-16T11:05:03.107517+0000","last_became_peered":"2024-03-16T11:05:03.107517+0000","last_unstale":"2024-03-16T11:05:31.990540+0000","last_undegraded":"2024-03-16T11:05:31.990540+0000","last_fullsized":"2024-03-16T11:05:31.990540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:07:51.828833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103117+0000","last_change":"2024-03-16T11:03:59.288822+0000","last_active":"2024-03-16T11:05:03.103117+0000","last_peered":"2024-03-16T11:05:03.103117+0000","last_clean":"2024-03-16T11:05:03.103117+0000","last_became_active":"2024-03-16T11:03:59.288516+0000","last_became_peered":"2024-03-16T11:03:59.288516+0000","last_unstale":"2024-03-16T11:05:03.103117+0000","last_undegraded":"2024-03-16T11:05:03.103117+0000","last_fullsized":"2024-03-16T11:05:03.103117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:32:38.862476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003380+0000","last_change":"2024-03-16T11:04:00.296117+0000","last_active":"2024-03-16T11:05:32.003380+0000","last_peered":"2024-03-16T11:05:32.003380+0000","last_clean":"2024-03-16T11:05:32.003380+0000","last_became_active":"2024-03-16T11:04:00.295664+0000","last_became_peered":"2024-03-16T11:04:00.295664+0000","last_unstale":"2024-03-16T11:05:32.003380+0000","last_undegraded":"2024-03-16T11:05:32.003380+0000","last_fullsized":"2024-03-16T11:05:32.003380+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.448616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999611+0000","last_change":"2024-03-16T11:04:00.295035+0000","last_active":"2024-03-16T11:05:31.999611+0000","last_peered":"2024-03-16T11:05:31.999611+0000","last_clean":"2024-03-16T11:05:31.999611+0000","last_became_active":"2024-03-16T11:04:00.294689+0000","last_became_peered":"2024-03-16T11:04:00.294689+0000","last_unstale":"2024-03-16T11:05:31.999611+0000","last_undegraded":"2024-03-16T11:05:31.999611+0000","last_fullsized":"2024-03-16T11:05:31.999611+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:55.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998519+0000","last_change":"2024-03-16T11:04:00.292831+0000","last_active":"2024-03-16T11:05:31.998519+0000","last_peered":"2024-03-16T11:05:31.998519+0000","last_clean":"2024-03-16T11:05:31.998519+0000","last_became_active":"2024-03-16T11:04:00.292387+0000","last_became_peered":"2024-03-16T11:04:00.292387+0000","last_unstale":"2024-03-16T11:05:31.998519+0000","last_undegraded":"2024-03-16T11:05:31.998519+0000","last_fullsized":"2024-03-16T11:05:31.998519+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:14:53.273535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.970952+0000","last_change":"2024-03-16T11:04:00.291267+0000","last_active":"2024-03-16T11:05:31.970952+0000","last_peered":"2024-03-16T11:05:31.970952+0000","last_clean":"2024-03-16T11:05:31.970952+0000","last_became_active":"2024-03-16T11:04:00.290911+0000","last_became_peered":"2024-03-16T11:04:00.290911+0000","last_unstale":"2024-03-16T11:05:31.970952+0000","last_undegraded":"2024-03-16T11:05:31.970952+0000","last_fullsized":"2024-03-16T11:05:31.970952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:35.285167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999212+0000","last_change":"2024-03-16T11:04:00.300470+0000","last_active":"2024-03-16T11:05:31.999212+0000","last_peered":"2024-03-16T11:05:31.999212+0000","last_clean":"2024-03-16T11:05:31.999212+0000","last_became_active":"2024-03-16T11:04:00.300107+0000","last_became_peered":"2024-03-16T11:04:00.300107+0000","last_unstale":"2024-03-16T11:05:31.999212+0000","last_undegraded":"2024-03-16T11:05:31.999212+0000","last_fullsized":"2024-03-16T11:05:31.999212+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:18.718749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.001171+0000","last_change":"2024-03-16T11:04:00.300857+0000","last_active":"2024-03-16T11:05:32.001171+0000","last_peered":"2024-03-16T11:05:32.001171+0000","last_clean":"2024-03-16T11:05:32.001171+0000","last_became_active":"2024-03-16T11:04:00.300551+0000","last_became_peered":"2024-03-16T11:04:00.300551+0000","last_unstale":"2024-03-16T11:05:32.001171+0000","last_undegraded":"2024-03-16T11:05:32.001171+0000","last_fullsized":"2024-03-16T11:05:32.001171+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:24.362561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.996929+0000","last_change":"2024-03-16T11:05:03.110781+0000","last_active":"2024-03-16T11:05:31.996929+0000","last_peered":"2024-03-16T11:05:31.996929+0000","last_clean":"2024-03-16T11:05:31.996929+0000","last_became_active":"2024-03-16T11:05:03.110375+0000","last_became_peered":"2024-03-16T11:05:03.110375+0000","last_unstale":"2024-03-16T11:05:31.996929+0000","last_undegraded":"2024-03-16T11:05:31.996929+0000","last_fullsized":"2024-03-16T11:05:31.996929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:28.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998128+0000","last_change":"2024-03-16T11:04:00.294943+0000","last_active":"2024-03-16T11:05:31.998128+0000","last_peered":"2024-03-16T11:05:31.998128+0000","last_clean":"2024-03-16T11:05:31.998128+0000","last_became_active":"2024-03-16T11:04:00.294637+0000","last_became_peered":"2024-03-16T11:04:00.294637+0000","last_unstale":"2024-03-16T11:05:31.998128+0000","last_undegraded":"2024-03-16T11:05:31.998128+0000","last_fullsized":"2024-03-16T11:05:31.998128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:03:09.716753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990054+0000","last_change":"2024-03-16T11:04:00.300737+0000","last_active":"2024-03-16T11:05:31.990054+0000","last_peered":"2024-03-16T11:05:31.990054+0000","last_clean":"2024-03-16T11:05:31.990054+0000","last_became_active":"2024-03-16T11:04:00.300410+0000","last_became_peered":"2024-03-16T11:04:00.300410+0000","last_unstale":"2024-03-16T11:05:31.990054+0000","last_undegraded":"2024-03-16T11:05:31.990054+0000","last_fullsized":"2024-03-16T11:05:31.990054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:18.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.971404+0000","last_change":"2024-03-16T11:04:00.292958+0000","last_active":"2024-03-16T11:05:31.971404+0000","last_peered":"2024-03-16T11:05:31.971404+0000","last_clean":"2024-03-16T11:05:31.971404+0000","last_became_active":"2024-03-16T11:04:00.292599+0000","last_became_peered":"2024-03-16T11:04:00.292599+0000","last_unstale":"2024-03-16T11:05:31.971404+0000","last_undegraded":"2024-03-16T11:05:31.971404+0000","last_fullsized":"2024-03-16T11:05:31.971404+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:46.370597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994683+0000","last_change":"2024-03-16T11:04:00.291381+0000","last_active":"2024-03-16T11:05:31.994683+0000","last_peered":"2024-03-16T11:05:31.994683+0000","last_clean":"2024-03-16T11:05:31.994683+0000","last_became_active":"2024-03-16T11:04:00.291046+0000","last_became_peered":"2024-03-16T11:04:00.291046+0000","last_unstale":"2024-03-16T11:05:31.994683+0000","last_undegraded":"2024-03-16T11:05:31.994683+0000","last_fullsized":"2024-03-16T11:05:31.994683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:08.185733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008021+0000","last_change":"2024-03-16T11:04:00.301381+0000","last_active":"2024-03-16T11:05:32.008021+0000","last_peered":"2024-03-16T11:05:32.008021+0000","last_clean":"2024-03-16T11:05:32.008021+0000","last_became_active":"2024-03-16T11:04:00.300990+0000","last_became_peered":"2024-03-16T11:04:00.300990+0000","last_unstale":"2024-03-16T11:05:32.008021+0000","last_undegraded":"2024-03-16T11:05:32.008021+0000","last_fullsized":"2024-03-16T11:05:32.008021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:07:43.775066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999944+0000","last_change":"2024-03-16T11:04:00.298220+0000","last_active":"2024-03-16T11:05:31.999944+0000","last_peered":"2024-03-16T11:05:31.999944+0000","last_clean":"2024-03-16T11:05:31.999944+0000","last_became_active":"2024-03-16T11:04:00.297776+0000","last_became_peered":"2024-03-16T11:04:00.297776+0000","last_unstale":"2024-03-16T11:05:31.999944+0000","last_undegraded":"2024-03-16T11:05:31.999944+0000","last_fullsized":"2024-03-16T11:05:31.999944+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:40.387309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999183+0000","last_change":"2024-03-16T11:04:00.292988+0000","last_active":"2024-03-16T11:05:31.999183+0000","last_peered":"2024-03-16T11:05:31.999183+0000","last_clean":"2024-03-16T11:05:31.999183+0000","last_became_active":"2024-03-16T11:04:00.292655+0000","last_became_peered":"2024-03-16T11:04:00.292655+0000","last_unstale":"2024-03-16T11:05:31.999183+0000","last_undegraded":"2024-03-16T11:05:31.999183+0000","last_fullsized":"2024-03-16T11:05:31.999183+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:33:43.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71813,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":702,"num_read_kb":424,"num_write":170,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":199303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285604,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":471093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547242,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1683456,"data_stored":1128327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743471,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":471307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005107,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":471205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1593344,"data_stored":1063298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430270,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":470813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":529650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":471967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":531000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":471007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":472158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1601536,"data_stored":1063217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:05:57.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T11:05:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:57 smithi153 ceph-mon[35521]: pgmap v347: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-16T11:05:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:57 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3293348003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:57 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/354591086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:57 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/768457151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 -- 172.21.15.78:0/3851754612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581018e0 msgr2=0x7fd958101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/3851754612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581018e0 0x7fd958101d60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd948009900 tx=0x7fd94802bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 -- 172.21.15.78:0/3851754612 shutdown_connections 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/3851754612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9581022a0 0x7fd958108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/3851754612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581018e0 0x7fd958101d60 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/3851754612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581006f0 0x7fd958100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 -- 172.21.15.78:0/3851754612 >> 172.21.15.78:0/3851754612 conn(0x7fd9580fbc80 msgr2=0x7fd9580fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 -- 172.21.15.78:0/3851754612 shutdown_connections 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.661+0000 7fd95f4dc640 1 -- 172.21.15.78:0/3851754612 wait complete. 2024-03-16T11:05:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 Processor -- start 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 -- start start 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9581006f0 0x7fd958196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581018e0 0x7fd958196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581022a0 0x7fd95819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd95810c220 con 0x7fd9581022a0 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd95810c0a0 con 0x7fd9581018e0 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd95810c3a0 con 0x7fd9581006f0 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581022a0 0x7fd95819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581022a0 0x7fd95819dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54800/0 (socket says 172.21.15.78:54800) 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95d278640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9581006f0 0x7fd958196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 -- 172.21.15.78:0/2753901607 learned_addr learned my addr 172.21.15.78:0/2753901607 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95ca77640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581018e0 0x7fd958196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 -- 172.21.15.78:0/2753901607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9581006f0 msgr2=0x7fd958196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9581006f0 0x7fd958196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 -- 172.21.15.78:0/2753901607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581018e0 msgr2=0x7fd958196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581018e0 0x7fd958196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 -- 172.21.15.78:0/2753901607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95819e530 con 0x7fd9581022a0 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95da79640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581022a0 0x7fd95819dec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd95000eb10 tx=0x7fd95000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd94e7fc640 1 -- 172.21.15.78:0/2753901607 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950004b20 con 0x7fd9581022a0 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd94e7fc640 1 -- 172.21.15.78:0/2753901607 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd950004cc0 con 0x7fd9581022a0 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd95819e820 con 0x7fd9581022a0 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.662+0000 7fd94e7fc640 1 -- 172.21.15.78:0/2753901607 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950010690 con 0x7fd9581022a0 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.663+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd95819ed40 con 0x7fd9581022a0 2024-03-16T11:05:57.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.663+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd958107a10 con 0x7fd9581022a0 2024-03-16T11:05:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.664+0000 7fd94e7fc640 1 -- 172.21.15.78:0/2753901607 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fd950005020 con 0x7fd9581022a0 2024-03-16T11:05:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.665+0000 7fd94e7fc640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd9440a2890 0x7fd9440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.665+0000 7fd95d278640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd9440a2890 0x7fd9440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.665+0000 7fd94e7fc640 1 -- 172.21.15.78:0/2753901607 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd9500cf2b0 con 0x7fd9581022a0 2024-03-16T11:05:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.666+0000 7fd95d278640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd9440a2890 0x7fd9440a4d50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd958101740 tx=0x7fd954005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:57.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.670+0000 7fd94e7fc640 1 -- 172.21.15.78:0/2753901607 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd95009c1f0 con 0x7fd9581022a0 2024-03-16T11:05:57.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.852+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd958061c20 con 0x7fd9440a2890 2024-03-16T11:05:57.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.859+0000 7fd94e7fc640 1 -- 172.21.15.78:0/2753901607 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258834 (secure 0 0 0) 0x7fd958061c20 con 0x7fd9440a2890 2024-03-16T11:05:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.862+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd9440a2890 msgr2=0x7fd9440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.862+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd9440a2890 0x7fd9440a4d50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd958101740 tx=0x7fd954005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.862+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581022a0 msgr2=0x7fd95819dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.862+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581022a0 0x7fd95819dec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd95000eb10 tx=0x7fd95000c2d0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 shutdown_connections 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9581022a0 0x7fd95819dec0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581018e0 0x7fd958196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd9440a2890 0x7fd9440a4d50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 --2- 172.21.15.78:0/2753901607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9581006f0 0x7fd958196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 >> 172.21.15.78:0/2753901607 conn(0x7fd9580fbc80 msgr2=0x7fd958104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 shutdown_connections 2024-03-16T11:05:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:57.863+0000 7fd95f4dc640 1 -- 172.21.15.78:0/2753901607 wait complete. 2024-03-16T11:05:57.874 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-03-16T11:05:55.993515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662415,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":828,"num_read_kb":533,"num_write":353,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383600,"kb_used_data":13648,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558224,"statfs":{"total":1151940427776,"available":1151547621376,"internally_reserved":0,"allocated":13975552,"data_stored":7615042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8510,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":49,"num_write":9,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002827"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008007+0000","last_change":"2024-03-16T11:04:00.295528+0000","last_active":"2024-03-16T11:05:32.008007+0000","last_peered":"2024-03-16T11:05:32.008007+0000","last_clean":"2024-03-16T11:05:32.008007+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:32.008007+0000","last_undegraded":"2024-03-16T11:05:32.008007+0000","last_fullsized":"2024-03-16T11:05:32.008007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:22.852488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994520+0000","last_change":"2024-03-16T11:04:00.289043+0000","last_active":"2024-03-16T11:05:31.994520+0000","last_peered":"2024-03-16T11:05:31.994520+0000","last_clean":"2024-03-16T11:05:31.994520+0000","last_became_active":"2024-03-16T11:04:00.288742+0000","last_became_peered":"2024-03-16T11:04:00.288742+0000","last_unstale":"2024-03-16T11:05:31.994520+0000","last_undegraded":"2024-03-16T11:05:31.994520+0000","last_fullsized":"2024-03-16T11:05:31.994520+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:07:56.710499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007950+0000","last_change":"2024-03-16T11:04:00.293443+0000","last_active":"2024-03-16T11:05:32.007950+0000","last_peered":"2024-03-16T11:05:32.007950+0000","last_clean":"2024-03-16T11:05:32.007950+0000","last_became_active":"2024-03-16T11:04:00.293117+0000","last_became_peered":"2024-03-16T11:04:00.293117+0000","last_unstale":"2024-03-16T11:05:32.007950+0000","last_undegraded":"2024-03-16T11:05:32.007950+0000","last_fullsized":"2024-03-16T11:05:32.007950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:49.728721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998576+0000","last_change":"2024-03-16T11:04:00.292105+0000","last_active":"2024-03-16T11:05:31.998576+0000","last_peered":"2024-03-16T11:05:31.998576+0000","last_clean":"2024-03-16T11:05:31.998576+0000","last_became_active":"2024-03-16T11:04:00.291755+0000","last_became_peered":"2024-03-16T11:04:00.291755+0000","last_unstale":"2024-03-16T11:05:31.998576+0000","last_undegraded":"2024-03-16T11:05:31.998576+0000","last_fullsized":"2024-03-16T11:05:31.998576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:48:43.734854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.997439+0000","last_change":"2024-03-16T11:04:00.282168+0000","last_active":"2024-03-16T11:05:31.997439+0000","last_peered":"2024-03-16T11:05:31.997439+0000","last_clean":"2024-03-16T11:05:31.997439+0000","last_became_active":"2024-03-16T11:04:00.281753+0000","last_became_peered":"2024-03-16T11:04:00.281753+0000","last_unstale":"2024-03-16T11:05:31.997439+0000","last_undegraded":"2024-03-16T11:05:31.997439+0000","last_fullsized":"2024-03-16T11:05:31.997439+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:13.223955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002045+0000","last_change":"2024-03-16T11:05:03.111462+0000","last_active":"2024-03-16T11:05:32.002045+0000","last_peered":"2024-03-16T11:05:32.002045+0000","last_clean":"2024-03-16T11:05:32.002045+0000","last_became_active":"2024-03-16T11:05:03.111087+0000","last_became_peered":"2024-03-16T11:05:03.111087+0000","last_unstale":"2024-03-16T11:05:32.002045+0000","last_undegraded":"2024-03-16T11:05:32.002045+0000","last_fullsized":"2024-03-16T11:05:32.002045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:09.339732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999489+0000","last_change":"2024-03-16T11:04:00.292446+0000","last_active":"2024-03-16T11:05:31.999489+0000","last_peered":"2024-03-16T11:05:31.999489+0000","last_clean":"2024-03-16T11:05:31.999489+0000","last_became_active":"2024-03-16T11:04:00.292079+0000","last_became_peered":"2024-03-16T11:04:00.292079+0000","last_unstale":"2024-03-16T11:05:31.999489+0000","last_undegraded":"2024-03-16T11:05:31.999489+0000","last_fullsized":"2024-03-16T11:05:31.999489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.325945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003952+0000","last_change":"2024-03-16T11:04:00.294644+0000","last_active":"2024-03-16T11:05:32.003952+0000","last_peered":"2024-03-16T11:05:32.003952+0000","last_clean":"2024-03-16T11:05:32.003952+0000","last_became_active":"2024-03-16T11:04:00.294283+0000","last_became_peered":"2024-03-16T11:04:00.294283+0000","last_unstale":"2024-03-16T11:05:32.003952+0000","last_undegraded":"2024-03-16T11:05:32.003952+0000","last_fullsized":"2024-03-16T11:05:32.003952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:43.125823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007001+0000","last_change":"2024-03-16T11:04:00.302512+0000","last_active":"2024-03-16T11:05:32.007001+0000","last_peered":"2024-03-16T11:05:32.007001+0000","last_clean":"2024-03-16T11:05:32.007001+0000","last_became_active":"2024-03-16T11:04:00.301891+0000","last_became_peered":"2024-03-16T11:04:00.301891+0000","last_unstale":"2024-03-16T11:05:32.007001+0000","last_undegraded":"2024-03-16T11:05:32.007001+0000","last_fullsized":"2024-03-16T11:05:32.007001+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:15.112295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003148+0000","last_change":"2024-03-16T11:04:00.293327+0000","last_active":"2024-03-16T11:05:32.003148+0000","last_peered":"2024-03-16T11:05:32.003148+0000","last_clean":"2024-03-16T11:05:32.003148+0000","last_became_active":"2024-03-16T11:04:00.292971+0000","last_became_peered":"2024-03-16T11:04:00.292971+0000","last_unstale":"2024-03-16T11:05:32.003148+0000","last_undegraded":"2024-03-16T11:05:32.003148+0000","last_fullsized":"2024-03-16T11:05:32.003148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:48:48.565861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003814+0000","last_change":"2024-03-16T11:04:00.298956+0000","last_active":"2024-03-16T11:05:32.003814+0000","last_peered":"2024-03-16T11:05:32.003814+0000","last_clean":"2024-03-16T11:05:32.003814+0000","last_became_active":"2024-03-16T11:04:00.298508+0000","last_became_peered":"2024-03-16T11:04:00.298508+0000","last_unstale":"2024-03-16T11:05:32.003814+0000","last_undegraded":"2024-03-16T11:05:32.003814+0000","last_fullsized":"2024-03-16T11:05:32.003814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:25:46.576514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002545+0000","last_change":"2024-03-16T11:04:00.294453+0000","last_active":"2024-03-16T11:05:32.002545+0000","last_peered":"2024-03-16T11:05:32.002545+0000","last_clean":"2024-03-16T11:05:32.002545+0000","last_became_active":"2024-03-16T11:04:00.294084+0000","last_became_peered":"2024-03-16T11:04:00.294084+0000","last_unstale":"2024-03-16T11:05:32.002545+0000","last_undegraded":"2024-03-16T11:05:32.002545+0000","last_fullsized":"2024-03-16T11:05:32.002545+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:12.924835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003591+0000","last_change":"2024-03-16T11:04:00.293149+0000","last_active":"2024-03-16T11:05:32.003591+0000","last_peered":"2024-03-16T11:05:32.003591+0000","last_clean":"2024-03-16T11:05:32.003591+0000","last_became_active":"2024-03-16T11:04:00.292839+0000","last_became_peered":"2024-03-16T11:04:00.292839+0000","last_unstale":"2024-03-16T11:05:32.003591+0000","last_undegraded":"2024-03-16T11:05:32.003591+0000","last_fullsized":"2024-03-16T11:05:32.003591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:34:38.446918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.993502+0000","last_change":"2024-03-16T11:04:00.285748+0000","last_active":"2024-03-16T11:05:31.993502+0000","last_peered":"2024-03-16T11:05:31.993502+0000","last_clean":"2024-03-16T11:05:31.993502+0000","last_became_active":"2024-03-16T11:04:00.285394+0000","last_became_peered":"2024-03-16T11:04:00.285394+0000","last_unstale":"2024-03-16T11:05:31.993502+0000","last_undegraded":"2024-03-16T11:05:31.993502+0000","last_fullsized":"2024-03-16T11:05:31.993502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:02.113271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999076+0000","last_change":"2024-03-16T11:04:00.301783+0000","last_active":"2024-03-16T11:05:31.999076+0000","last_peered":"2024-03-16T11:05:31.999076+0000","last_clean":"2024-03-16T11:05:31.999076+0000","last_became_active":"2024-03-16T11:04:00.301313+0000","last_became_peered":"2024-03-16T11:04:00.301313+0000","last_unstale":"2024-03-16T11:05:31.999076+0000","last_undegraded":"2024-03-16T11:05:31.999076+0000","last_fullsized":"2024-03-16T11:05:31.999076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:19:32.243602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007419+0000","last_change":"2024-03-16T11:04:00.301375+0000","last_active":"2024-03-16T11:05:32.007419+0000","last_peered":"2024-03-16T11:05:32.007419+0000","last_clean":"2024-03-16T11:05:32.007419+0000","last_became_active":"2024-03-16T11:04:00.301013+0000","last_became_peered":"2024-03-16T11:04:00.301013+0000","last_unstale":"2024-03-16T11:05:32.007419+0000","last_undegraded":"2024-03-16T11:05:32.007419+0000","last_fullsized":"2024-03-16T11:05:32.007419+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:21:52.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008074+0000","last_change":"2024-03-16T11:04:00.296287+0000","last_active":"2024-03-16T11:05:32.008074+0000","last_peered":"2024-03-16T11:05:32.008074+0000","last_clean":"2024-03-16T11:05:32.008074+0000","last_became_active":"2024-03-16T11:04:00.295972+0000","last_became_peered":"2024-03-16T11:04:00.295972+0000","last_unstale":"2024-03-16T11:05:32.008074+0000","last_undegraded":"2024-03-16T11:05:32.008074+0000","last_fullsized":"2024-03-16T11:05:32.008074+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:14.586052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006625+0000","last_change":"2024-03-16T11:04:00.295551+0000","last_active":"2024-03-16T11:05:32.006625+0000","last_peered":"2024-03-16T11:05:32.006625+0000","last_clean":"2024-03-16T11:05:32.006625+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:32.006625+0000","last_undegraded":"2024-03-16T11:05:32.006625+0000","last_fullsized":"2024-03-16T11:05:32.006625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:53.495224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.971033+0000","last_change":"2024-03-16T11:04:00.290455+0000","last_active":"2024-03-16T11:05:31.971033+0000","last_peered":"2024-03-16T11:05:31.971033+0000","last_clean":"2024-03-16T11:05:31.971033+0000","last_became_active":"2024-03-16T11:04:00.290057+0000","last_became_peered":"2024-03-16T11:04:00.290057+0000","last_unstale":"2024-03-16T11:05:31.971033+0000","last_undegraded":"2024-03-16T11:05:31.971033+0000","last_fullsized":"2024-03-16T11:05:31.971033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:57:43.677050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102676+0000","last_change":"2024-03-16T11:03:59.285059+0000","last_active":"2024-03-16T11:05:03.102676+0000","last_peered":"2024-03-16T11:05:03.102676+0000","last_clean":"2024-03-16T11:05:03.102676+0000","last_became_active":"2024-03-16T11:03:59.284745+0000","last_became_peered":"2024-03-16T11:03:59.284745+0000","last_unstale":"2024-03-16T11:05:03.102676+0000","last_undegraded":"2024-03-16T11:05:03.102676+0000","last_fullsized":"2024-03-16T11:05:03.102676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:12.516958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990186+0000","last_change":"2024-03-16T11:04:00.292933+0000","last_active":"2024-03-16T11:05:31.990186+0000","last_peered":"2024-03-16T11:05:31.990186+0000","last_clean":"2024-03-16T11:05:31.990186+0000","last_became_active":"2024-03-16T11:04:00.292637+0000","last_became_peered":"2024-03-16T11:04:00.292637+0000","last_unstale":"2024-03-16T11:05:31.990186+0000","last_undegraded":"2024-03-16T11:05:31.990186+0000","last_fullsized":"2024-03-16T11:05:31.990186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:22.231735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107896+0000","last_change":"2024-03-16T11:03:59.285267+0000","last_active":"2024-03-16T11:05:03.107896+0000","last_peered":"2024-03-16T11:05:03.107896+0000","last_clean":"2024-03-16T11:05:03.107896+0000","last_became_active":"2024-03-16T11:03:59.284890+0000","last_became_peered":"2024-03-16T11:03:59.284890+0000","last_unstale":"2024-03-16T11:05:03.107896+0000","last_undegraded":"2024-03-16T11:05:03.107896+0000","last_fullsized":"2024-03-16T11:05:03.107896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:53.625612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008052+0000","last_change":"2024-03-16T11:04:00.292660+0000","last_active":"2024-03-16T11:05:32.008052+0000","last_peered":"2024-03-16T11:05:32.008052+0000","last_clean":"2024-03-16T11:05:32.008052+0000","last_became_active":"2024-03-16T11:04:00.292366+0000","last_became_peered":"2024-03-16T11:04:00.292366+0000","last_unstale":"2024-03-16T11:05:32.008052+0000","last_undegraded":"2024-03-16T11:05:32.008052+0000","last_fullsized":"2024-03-16T11:05:32.008052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:36:44.331456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109296+0000","last_change":"2024-03-16T11:03:59.286370+0000","last_active":"2024-03-16T11:05:03.109296+0000","last_peered":"2024-03-16T11:05:03.109296+0000","last_clean":"2024-03-16T11:05:03.109296+0000","last_became_active":"2024-03-16T11:03:59.286065+0000","last_became_peered":"2024-03-16T11:03:59.286065+0000","last_unstale":"2024-03-16T11:05:03.109296+0000","last_undegraded":"2024-03-16T11:05:03.109296+0000","last_fullsized":"2024-03-16T11:05:03.109296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:20.126848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007484+0000","last_change":"2024-03-16T11:04:00.290864+0000","last_active":"2024-03-16T11:05:32.007484+0000","last_peered":"2024-03-16T11:05:32.007484+0000","last_clean":"2024-03-16T11:05:32.007484+0000","last_became_active":"2024-03-16T11:04:00.290537+0000","last_became_peered":"2024-03-16T11:04:00.290537+0000","last_unstale":"2024-03-16T11:05:32.007484+0000","last_undegraded":"2024-03-16T11:05:32.007484+0000","last_fullsized":"2024-03-16T11:05:32.007484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:52.991767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109155+0000","last_change":"2024-03-16T11:03:59.281663+0000","last_active":"2024-03-16T11:05:03.109155+0000","last_peered":"2024-03-16T11:05:03.109155+0000","last_clean":"2024-03-16T11:05:03.109155+0000","last_became_active":"2024-03-16T11:03:59.281328+0000","last_became_peered":"2024-03-16T11:03:59.281328+0000","last_unstale":"2024-03-16T11:05:03.109155+0000","last_undegraded":"2024-03-16T11:05:03.109155+0000","last_fullsized":"2024-03-16T11:05:03.109155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:44.511109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003970+0000","last_change":"2024-03-16T11:05:03.106586+0000","last_active":"2024-03-16T11:05:32.003970+0000","last_peered":"2024-03-16T11:05:32.003970+0000","last_clean":"2024-03-16T11:05:32.003970+0000","last_became_active":"2024-03-16T11:05:03.106233+0000","last_became_peered":"2024-03-16T11:05:03.106233+0000","last_unstale":"2024-03-16T11:05:32.003970+0000","last_undegraded":"2024-03-16T11:05:32.003970+0000","last_fullsized":"2024-03-16T11:05:32.003970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:19:30.480318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":142,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:54.588491+0000","last_change":"2024-03-16T11:03:59.286300+0000","last_active":"2024-03-16T11:05:54.588491+0000","last_peered":"2024-03-16T11:05:54.588491+0000","last_clean":"2024-03-16T11:05:54.588491+0000","last_became_active":"2024-03-16T11:03:59.285764+0000","last_became_peered":"2024-03-16T11:03:59.285764+0000","last_unstale":"2024-03-16T11:05:54.588491+0000","last_undegraded":"2024-03-16T11:05:54.588491+0000","last_fullsized":"2024-03-16T11:05:54.588491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:49.084245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.984380+0000","last_change":"2024-03-16T11:04:00.289152+0000","last_active":"2024-03-16T11:05:31.984380+0000","last_peered":"2024-03-16T11:05:31.984380+0000","last_clean":"2024-03-16T11:05:31.984380+0000","last_became_active":"2024-03-16T11:04:00.288837+0000","last_became_peered":"2024-03-16T11:04:00.288837+0000","last_unstale":"2024-03-16T11:05:31.984380+0000","last_undegraded":"2024-03-16T11:05:31.984380+0000","last_fullsized":"2024-03-16T11:05:31.984380+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:45:01.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106073+0000","last_change":"2024-03-16T11:03:59.281505+0000","last_active":"2024-03-16T11:05:03.106073+0000","last_peered":"2024-03-16T11:05:03.106073+0000","last_clean":"2024-03-16T11:05:03.106073+0000","last_became_active":"2024-03-16T11:03:59.281175+0000","last_became_peered":"2024-03-16T11:03:59.281175+0000","last_unstale":"2024-03-16T11:05:03.106073+0000","last_undegraded":"2024-03-16T11:05:03.106073+0000","last_fullsized":"2024-03-16T11:05:03.106073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:12.463914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.988807+0000","last_change":"2024-03-16T11:04:00.297769+0000","last_active":"2024-03-16T11:05:31.988807+0000","last_peered":"2024-03-16T11:05:31.988807+0000","last_clean":"2024-03-16T11:05:31.988807+0000","last_became_active":"2024-03-16T11:04:00.297061+0000","last_became_peered":"2024-03-16T11:04:00.297061+0000","last_unstale":"2024-03-16T11:05:31.988807+0000","last_undegraded":"2024-03-16T11:05:31.988807+0000","last_fullsized":"2024-03-16T11:05:31.988807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:42.594315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103214+0000","last_change":"2024-03-16T11:03:59.290260+0000","last_active":"2024-03-16T11:05:03.103214+0000","last_peered":"2024-03-16T11:05:03.103214+0000","last_clean":"2024-03-16T11:05:03.103214+0000","last_became_active":"2024-03-16T11:03:59.289843+0000","last_became_peered":"2024-03-16T11:03:59.289843+0000","last_unstale":"2024-03-16T11:05:03.103214+0000","last_undegraded":"2024-03-16T11:05:03.103214+0000","last_fullsized":"2024-03-16T11:05:03.103214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:22.483488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.997150+0000","last_change":"2024-03-16T11:04:00.287873+0000","last_active":"2024-03-16T11:05:31.997150+0000","last_peered":"2024-03-16T11:05:31.997150+0000","last_clean":"2024-03-16T11:05:31.997150+0000","last_became_active":"2024-03-16T11:04:00.287002+0000","last_became_peered":"2024-03-16T11:04:00.287002+0000","last_unstale":"2024-03-16T11:05:31.997150+0000","last_undegraded":"2024-03-16T11:05:31.997150+0000","last_fullsized":"2024-03-16T11:05:31.997150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:57.851726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102815+0000","last_change":"2024-03-16T11:03:59.284745+0000","last_active":"2024-03-16T11:05:03.102815+0000","last_peered":"2024-03-16T11:05:03.102815+0000","last_clean":"2024-03-16T11:05:03.102815+0000","last_became_active":"2024-03-16T11:03:59.284325+0000","last_became_peered":"2024-03-16T11:03:59.284325+0000","last_unstale":"2024-03-16T11:05:03.102815+0000","last_undegraded":"2024-03-16T11:05:03.102815+0000","last_fullsized":"2024-03-16T11:05:03.102815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.537787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002828+0000","last_change":"2024-03-16T11:04:00.290698+0000","last_active":"2024-03-16T11:05:32.002828+0000","last_peered":"2024-03-16T11:05:32.002828+0000","last_clean":"2024-03-16T11:05:32.002828+0000","last_became_active":"2024-03-16T11:04:00.290348+0000","last_became_peered":"2024-03-16T11:04:00.290348+0000","last_unstale":"2024-03-16T11:05:32.002828+0000","last_undegraded":"2024-03-16T11:05:32.002828+0000","last_fullsized":"2024-03-16T11:05:32.002828+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:44.114305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":135,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.586690+0000","last_change":"2024-03-16T11:03:59.280705+0000","last_active":"2024-03-16T11:05:53.586690+0000","last_peered":"2024-03-16T11:05:53.586690+0000","last_clean":"2024-03-16T11:05:53.586690+0000","last_became_active":"2024-03-16T11:03:59.280372+0000","last_became_peered":"2024-03-16T11:03:59.280372+0000","last_unstale":"2024-03-16T11:05:53.586690+0000","last_undegraded":"2024-03-16T11:05:53.586690+0000","last_fullsized":"2024-03-16T11:05:53.586690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:54.187322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006386+0000","last_change":"2024-03-16T11:04:00.293867+0000","last_active":"2024-03-16T11:05:32.006386+0000","last_peered":"2024-03-16T11:05:32.006386+0000","last_clean":"2024-03-16T11:05:32.006386+0000","last_became_active":"2024-03-16T11:04:00.293330+0000","last_became_peered":"2024-03-16T11:04:00.293330+0000","last_unstale":"2024-03-16T11:05:32.006386+0000","last_undegraded":"2024-03-16T11:05:32.006386+0000","last_fullsized":"2024-03-16T11:05:32.006386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:54.798346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105547+0000","last_change":"2024-03-16T11:03:59.284202+0000","last_active":"2024-03-16T11:05:03.105547+0000","last_peered":"2024-03-16T11:05:03.105547+0000","last_clean":"2024-03-16T11:05:03.105547+0000","last_became_active":"2024-03-16T11:03:59.283656+0000","last_became_peered":"2024-03-16T11:03:59.283656+0000","last_unstale":"2024-03-16T11:05:03.105547+0000","last_undegraded":"2024-03-16T11:05:03.105547+0000","last_fullsized":"2024-03-16T11:05:03.105547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:44.760307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.996327+0000","last_change":"2024-03-16T11:05:03.111443+0000","last_active":"2024-03-16T11:05:31.996327+0000","last_peered":"2024-03-16T11:05:31.996327+0000","last_clean":"2024-03-16T11:05:31.996327+0000","last_became_active":"2024-03-16T11:05:03.111027+0000","last_became_peered":"2024-03-16T11:05:03.111027+0000","last_unstale":"2024-03-16T11:05:31.996327+0000","last_undegraded":"2024-03-16T11:05:31.996327+0000","last_fullsized":"2024-03-16T11:05:31.996327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:31.844283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103330+0000","last_change":"2024-03-16T11:03:59.288268+0000","last_active":"2024-03-16T11:05:03.103330+0000","last_peered":"2024-03-16T11:05:03.103330+0000","last_clean":"2024-03-16T11:05:03.103330+0000","last_became_active":"2024-03-16T11:03:59.287861+0000","last_became_peered":"2024-03-16T11:03:59.287861+0000","last_unstale":"2024-03-16T11:05:03.103330+0000","last_undegraded":"2024-03-16T11:05:03.103330+0000","last_fullsized":"2024-03-16T11:05:03.103330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:34:34.248408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007936+0000","last_change":"2024-03-16T11:04:00.306490+0000","last_active":"2024-03-16T11:05:32.007936+0000","last_peered":"2024-03-16T11:05:32.007936+0000","last_clean":"2024-03-16T11:05:32.007936+0000","last_became_active":"2024-03-16T11:04:00.305874+0000","last_became_peered":"2024-03-16T11:04:00.305874+0000","last_unstale":"2024-03-16T11:05:32.007936+0000","last_undegraded":"2024-03-16T11:05:32.007936+0000","last_fullsized":"2024-03-16T11:05:32.007936+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":286,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:54.589113+0000","last_change":"2024-03-16T11:03:59.280714+0000","last_active":"2024-03-16T11:05:54.589113+0000","last_peered":"2024-03-16T11:05:54.589113+0000","last_clean":"2024-03-16T11:05:54.589113+0000","last_became_active":"2024-03-16T11:03:59.280398+0000","last_became_peered":"2024-03-16T11:03:59.280398+0000","last_unstale":"2024-03-16T11:05:54.589113+0000","last_undegraded":"2024-03-16T11:05:54.589113+0000","last_fullsized":"2024-03-16T11:05:54.589113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:34.256125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66294,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":234,"num_read_kb":179,"num_write":34,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999739+0000","last_change":"2024-03-16T11:04:00.285669+0000","last_active":"2024-03-16T11:05:31.999739+0000","last_peered":"2024-03-16T11:05:31.999739+0000","last_clean":"2024-03-16T11:05:31.999739+0000","last_became_active":"2024-03-16T11:04:00.285337+0000","last_became_peered":"2024-03-16T11:04:00.285337+0000","last_unstale":"2024-03-16T11:05:31.999739+0000","last_undegraded":"2024-03-16T11:05:31.999739+0000","last_fullsized":"2024-03-16T11:05:31.999739+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:36.925350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548506+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548506+0000","last_peered":"2024-03-16T11:05:03.548506+0000","last_clean":"2024-03-16T11:05:03.548506+0000","last_became_active":"2024-03-16T11:03:59.289357+0000","last_became_peered":"2024-03-16T11:03:59.289357+0000","last_unstale":"2024-03-16T11:05:03.548506+0000","last_undegraded":"2024-03-16T11:05:03.548506+0000","last_fullsized":"2024-03-16T11:05:03.548506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:26:35.860091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998250+0000","last_change":"2024-03-16T11:04:00.298488+0000","last_active":"2024-03-16T11:05:31.998250+0000","last_peered":"2024-03-16T11:05:31.998250+0000","last_clean":"2024-03-16T11:05:31.998250+0000","last_became_active":"2024-03-16T11:04:00.298202+0000","last_became_peered":"2024-03-16T11:04:00.298202+0000","last_unstale":"2024-03-16T11:05:31.998250+0000","last_undegraded":"2024-03-16T11:05:31.998250+0000","last_fullsized":"2024-03-16T11:05:31.998250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:48.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102961+0000","last_change":"2024-03-16T11:03:59.287066+0000","last_active":"2024-03-16T11:05:03.102961+0000","last_peered":"2024-03-16T11:05:03.102961+0000","last_clean":"2024-03-16T11:05:03.102961+0000","last_became_active":"2024-03-16T11:03:59.286731+0000","last_became_peered":"2024-03-16T11:03:59.286731+0000","last_unstale":"2024-03-16T11:05:03.102961+0000","last_undegraded":"2024-03-16T11:05:03.102961+0000","last_fullsized":"2024-03-16T11:05:03.102961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:51.443833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999855+0000","last_change":"2024-03-16T11:04:00.294450+0000","last_active":"2024-03-16T11:05:31.999855+0000","last_peered":"2024-03-16T11:05:31.999855+0000","last_clean":"2024-03-16T11:05:31.999855+0000","last_became_active":"2024-03-16T11:04:00.293769+0000","last_became_peered":"2024-03-16T11:04:00.293769+0000","last_unstale":"2024-03-16T11:05:31.999855+0000","last_undegraded":"2024-03-16T11:05:31.999855+0000","last_fullsized":"2024-03-16T11:05:31.999855+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:02.755907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":141,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.597588+0000","last_change":"2024-03-16T11:03:59.286197+0000","last_active":"2024-03-16T11:05:53.597588+0000","last_peered":"2024-03-16T11:05:53.597588+0000","last_clean":"2024-03-16T11:05:53.597588+0000","last_became_active":"2024-03-16T11:03:59.285757+0000","last_became_peered":"2024-03-16T11:03:59.285757+0000","last_unstale":"2024-03-16T11:05:53.597588+0000","last_undegraded":"2024-03-16T11:05:53.597588+0000","last_fullsized":"2024-03-16T11:05:53.597588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:47.868106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002798+0000","last_change":"2024-03-16T11:04:00.296709+0000","last_active":"2024-03-16T11:05:32.002798+0000","last_peered":"2024-03-16T11:05:32.002798+0000","last_clean":"2024-03-16T11:05:32.002798+0000","last_became_active":"2024-03-16T11:04:00.296372+0000","last_became_peered":"2024-03-16T11:04:00.296372+0000","last_unstale":"2024-03-16T11:05:32.002798+0000","last_undegraded":"2024-03-16T11:05:32.002798+0000","last_fullsized":"2024-03-16T11:05:32.002798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:37.948866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108997+0000","last_change":"2024-03-16T11:03:59.283831+0000","last_active":"2024-03-16T11:05:03.108997+0000","last_peered":"2024-03-16T11:05:03.108997+0000","last_clean":"2024-03-16T11:05:03.108997+0000","last_became_active":"2024-03-16T11:03:59.283404+0000","last_became_peered":"2024-03-16T11:03:59.283404+0000","last_unstale":"2024-03-16T11:05:03.108997+0000","last_undegraded":"2024-03-16T11:05:03.108997+0000","last_fullsized":"2024-03-16T11:05:03.108997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:26:34.012597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.984955+0000","last_change":"2024-03-16T11:04:00.299703+0000","last_active":"2024-03-16T11:05:31.984955+0000","last_peered":"2024-03-16T11:05:31.984955+0000","last_clean":"2024-03-16T11:05:31.984955+0000","last_became_active":"2024-03-16T11:04:00.299321+0000","last_became_peered":"2024-03-16T11:04:00.299321+0000","last_unstale":"2024-03-16T11:05:31.984955+0000","last_undegraded":"2024-03-16T11:05:31.984955+0000","last_fullsized":"2024-03-16T11:05:31.984955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.529281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109510+0000","last_change":"2024-03-16T11:03:59.277837+0000","last_active":"2024-03-16T11:05:03.109510+0000","last_peered":"2024-03-16T11:05:03.109510+0000","last_clean":"2024-03-16T11:05:03.109510+0000","last_became_active":"2024-03-16T11:03:59.277515+0000","last_became_peered":"2024-03-16T11:03:59.277515+0000","last_unstale":"2024-03-16T11:05:03.109510+0000","last_undegraded":"2024-03-16T11:05:03.109510+0000","last_fullsized":"2024-03-16T11:05:03.109510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:05.116990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994100+0000","last_change":"2024-03-16T11:04:00.292998+0000","last_active":"2024-03-16T11:05:31.994100+0000","last_peered":"2024-03-16T11:05:31.994100+0000","last_clean":"2024-03-16T11:05:31.994100+0000","last_became_active":"2024-03-16T11:04:00.292608+0000","last_became_peered":"2024-03-16T11:04:00.292608+0000","last_unstale":"2024-03-16T11:05:31.994100+0000","last_undegraded":"2024-03-16T11:05:31.994100+0000","last_fullsized":"2024-03-16T11:05:31.994100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:41.996826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108505+0000","last_change":"2024-03-16T11:05:03.108505+0000","last_active":"2024-03-16T11:05:03.108505+0000","last_peered":"2024-03-16T11:05:03.108505+0000","last_clean":"2024-03-16T11:05:03.108505+0000","last_became_active":"2024-03-16T11:05:03.108132+0000","last_became_peered":"2024-03-16T11:05:03.108132+0000","last_unstale":"2024-03-16T11:05:03.108505+0000","last_undegraded":"2024-03-16T11:05:03.108505+0000","last_fullsized":"2024-03-16T11:05:03.108505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:50.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002870+0000","last_change":"2024-03-16T11:04:00.288897+0000","last_active":"2024-03-16T11:05:32.002870+0000","last_peered":"2024-03-16T11:05:32.002870+0000","last_clean":"2024-03-16T11:05:32.002870+0000","last_became_active":"2024-03-16T11:04:00.288493+0000","last_became_peered":"2024-03-16T11:04:00.288493+0000","last_unstale":"2024-03-16T11:05:32.002870+0000","last_undegraded":"2024-03-16T11:05:32.002870+0000","last_fullsized":"2024-03-16T11:05:32.002870+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:31.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105515+0000","last_change":"2024-03-16T11:03:59.291595+0000","last_active":"2024-03-16T11:05:03.105515+0000","last_peered":"2024-03-16T11:05:03.105515+0000","last_clean":"2024-03-16T11:05:03.105515+0000","last_became_active":"2024-03-16T11:03:59.291159+0000","last_became_peered":"2024-03-16T11:03:59.291159+0000","last_unstale":"2024-03-16T11:05:03.105515+0000","last_undegraded":"2024-03-16T11:05:03.105515+0000","last_fullsized":"2024-03-16T11:05:03.105515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:03.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000305+0000","last_change":"2024-03-16T11:05:03.108818+0000","last_active":"2024-03-16T11:05:32.000305+0000","last_peered":"2024-03-16T11:05:32.000305+0000","last_clean":"2024-03-16T11:05:32.000305+0000","last_became_active":"2024-03-16T11:05:03.108479+0000","last_became_peered":"2024-03-16T11:05:03.108479+0000","last_unstale":"2024-03-16T11:05:32.000305+0000","last_undegraded":"2024-03-16T11:05:32.000305+0000","last_fullsized":"2024-03-16T11:05:32.000305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:10:11.329743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.977502+0000","last_change":"2024-03-16T11:04:00.298400+0000","last_active":"2024-03-16T11:05:31.977502+0000","last_peered":"2024-03-16T11:05:31.977502+0000","last_clean":"2024-03-16T11:05:31.977502+0000","last_became_active":"2024-03-16T11:04:00.298026+0000","last_became_peered":"2024-03-16T11:04:00.298026+0000","last_unstale":"2024-03-16T11:05:31.977502+0000","last_undegraded":"2024-03-16T11:05:31.977502+0000","last_fullsized":"2024-03-16T11:05:31.977502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:08.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":136,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.225688+0000","last_change":"2024-03-16T11:03:59.287528+0000","last_active":"2024-03-16T11:05:53.225688+0000","last_peered":"2024-03-16T11:05:53.225688+0000","last_clean":"2024-03-16T11:05:53.225688+0000","last_became_active":"2024-03-16T11:03:59.287218+0000","last_became_peered":"2024-03-16T11:03:59.287218+0000","last_unstale":"2024-03-16T11:05:53.225688+0000","last_undegraded":"2024-03-16T11:05:53.225688+0000","last_fullsized":"2024-03-16T11:05:53.225688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:19.803233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006796+0000","last_change":"2024-03-16T11:05:03.106099+0000","last_active":"2024-03-16T11:05:32.006796+0000","last_peered":"2024-03-16T11:05:32.006796+0000","last_clean":"2024-03-16T11:05:32.006796+0000","last_became_active":"2024-03-16T11:05:03.105412+0000","last_became_peered":"2024-03-16T11:05:03.105412+0000","last_unstale":"2024-03-16T11:05:32.006796+0000","last_undegraded":"2024-03-16T11:05:32.006796+0000","last_fullsized":"2024-03-16T11:05:32.006796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:29:33.213676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.541779+0000","last_change":"2024-03-16T11:03:59.281251+0000","last_active":"2024-03-16T11:05:03.541779+0000","last_peered":"2024-03-16T11:05:03.541779+0000","last_clean":"2024-03-16T11:05:03.541779+0000","last_became_active":"2024-03-16T11:03:59.280830+0000","last_became_peered":"2024-03-16T11:03:59.280830+0000","last_unstale":"2024-03-16T11:05:03.541779+0000","last_undegraded":"2024-03-16T11:05:03.541779+0000","last_fullsized":"2024-03-16T11:05:03.541779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:03:04.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105473+0000","last_change":"2024-03-16T11:02:48.497749+0000","last_active":"2024-03-16T11:05:03.105473+0000","last_peered":"2024-03-16T11:05:03.105473+0000","last_clean":"2024-03-16T11:05:03.105473+0000","last_became_active":"2024-03-16T11:02:48.497157+0000","last_became_peered":"2024-03-16T11:02:48.497157+0000","last_unstale":"2024-03-16T11:05:03.105473+0000","last_undegraded":"2024-03-16T11:05:03.105473+0000","last_fullsized":"2024-03-16T11:05:03.105473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_clean_scrub_stamp":"2024-03-16T10:59:26.309639+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:04.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999743+0000","last_change":"2024-03-16T11:04:00.300821+0000","last_active":"2024-03-16T11:05:31.999743+0000","last_peered":"2024-03-16T11:05:31.999743+0000","last_clean":"2024-03-16T11:05:31.999743+0000","last_became_active":"2024-03-16T11:04:00.300510+0000","last_became_peered":"2024-03-16T11:04:00.300510+0000","last_unstale":"2024-03-16T11:05:31.999743+0000","last_undegraded":"2024-03-16T11:05:31.999743+0000","last_fullsized":"2024-03-16T11:05:31.999743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:57.035167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105571+0000","last_change":"2024-03-16T11:03:59.283609+0000","last_active":"2024-03-16T11:05:03.105571+0000","last_peered":"2024-03-16T11:05:03.105571+0000","last_clean":"2024-03-16T11:05:03.105571+0000","last_became_active":"2024-03-16T11:03:59.283227+0000","last_became_peered":"2024-03-16T11:03:59.283227+0000","last_unstale":"2024-03-16T11:05:03.105571+0000","last_undegraded":"2024-03-16T11:05:03.105571+0000","last_fullsized":"2024-03-16T11:05:03.105571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:18.904202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.112457+0000","last_change":"2024-03-16T11:05:03.112071+0000","last_active":"2024-03-16T11:05:03.112457+0000","last_peered":"2024-03-16T11:05:03.112457+0000","last_clean":"2024-03-16T11:05:03.112457+0000","last_became_active":"2024-03-16T11:05:03.108434+0000","last_became_peered":"2024-03-16T11:05:03.108434+0000","last_unstale":"2024-03-16T11:05:03.112457+0000","last_undegraded":"2024-03-16T11:05:03.112457+0000","last_fullsized":"2024-03-16T11:05:03.112457+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:24.758886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000550+0000","last_change":"2024-03-16T11:04:00.285134+0000","last_active":"2024-03-16T11:05:32.000550+0000","last_peered":"2024-03-16T11:05:32.000550+0000","last_clean":"2024-03-16T11:05:32.000550+0000","last_became_active":"2024-03-16T11:04:00.284705+0000","last_became_peered":"2024-03-16T11:04:00.284705+0000","last_unstale":"2024-03-16T11:05:32.000550+0000","last_undegraded":"2024-03-16T11:05:32.000550+0000","last_fullsized":"2024-03-16T11:05:32.000550+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:38.637970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102785+0000","last_change":"2024-03-16T11:03:59.289999+0000","last_active":"2024-03-16T11:05:03.102785+0000","last_peered":"2024-03-16T11:05:03.102785+0000","last_clean":"2024-03-16T11:05:03.102785+0000","last_became_active":"2024-03-16T11:03:59.289694+0000","last_became_peered":"2024-03-16T11:03:59.289694+0000","last_unstale":"2024-03-16T11:05:03.102785+0000","last_undegraded":"2024-03-16T11:05:03.102785+0000","last_fullsized":"2024-03-16T11:05:03.102785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:16.037387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000119+0000","last_change":"2024-03-16T11:04:00.298148+0000","last_active":"2024-03-16T11:05:32.000119+0000","last_peered":"2024-03-16T11:05:32.000119+0000","last_clean":"2024-03-16T11:05:32.000119+0000","last_became_active":"2024-03-16T11:04:00.297822+0000","last_became_peered":"2024-03-16T11:04:00.297822+0000","last_unstale":"2024-03-16T11:05:32.000119+0000","last_undegraded":"2024-03-16T11:05:32.000119+0000","last_fullsized":"2024-03-16T11:05:32.000119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:09:28.645317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542005+0000","last_change":"2024-03-16T11:03:59.292207+0000","last_active":"2024-03-16T11:05:03.542005+0000","last_peered":"2024-03-16T11:05:03.542005+0000","last_clean":"2024-03-16T11:05:03.542005+0000","last_became_active":"2024-03-16T11:03:59.291859+0000","last_became_peered":"2024-03-16T11:03:59.291859+0000","last_unstale":"2024-03-16T11:05:03.542005+0000","last_undegraded":"2024-03-16T11:05:03.542005+0000","last_fullsized":"2024-03-16T11:05:03.542005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:55.365720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998794+0000","last_change":"2024-03-16T11:04:00.291261+0000","last_active":"2024-03-16T11:05:31.998794+0000","last_peered":"2024-03-16T11:05:31.998794+0000","last_clean":"2024-03-16T11:05:31.998794+0000","last_became_active":"2024-03-16T11:04:00.290832+0000","last_became_peered":"2024-03-16T11:04:00.290832+0000","last_unstale":"2024-03-16T11:05:31.998794+0000","last_undegraded":"2024-03-16T11:05:31.998794+0000","last_fullsized":"2024-03-16T11:05:31.998794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:10.418808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.100986+0000","last_change":"2024-03-16T11:03:59.288014+0000","last_active":"2024-03-16T11:05:03.100986+0000","last_peered":"2024-03-16T11:05:03.100986+0000","last_clean":"2024-03-16T11:05:03.100986+0000","last_became_active":"2024-03-16T11:03:59.287679+0000","last_became_peered":"2024-03-16T11:03:59.287679+0000","last_unstale":"2024-03-16T11:05:03.100986+0000","last_undegraded":"2024-03-16T11:05:03.100986+0000","last_fullsized":"2024-03-16T11:05:03.100986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:16:39.156224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.960063+0000","last_change":"2024-03-16T11:04:00.288047+0000","last_active":"2024-03-16T11:05:31.960063+0000","last_peered":"2024-03-16T11:05:31.960063+0000","last_clean":"2024-03-16T11:05:31.960063+0000","last_became_active":"2024-03-16T11:04:00.287619+0000","last_became_peered":"2024-03-16T11:04:00.287619+0000","last_unstale":"2024-03-16T11:05:31.960063+0000","last_undegraded":"2024-03-16T11:05:31.960063+0000","last_fullsized":"2024-03-16T11:05:31.960063+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:43.608349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106759+0000","last_change":"2024-03-16T11:03:59.288439+0000","last_active":"2024-03-16T11:05:03.106759+0000","last_peered":"2024-03-16T11:05:03.106759+0000","last_clean":"2024-03-16T11:05:03.106759+0000","last_became_active":"2024-03-16T11:03:59.288122+0000","last_became_peered":"2024-03-16T11:03:59.288122+0000","last_unstale":"2024-03-16T11:05:03.106759+0000","last_undegraded":"2024-03-16T11:05:03.106759+0000","last_fullsized":"2024-03-16T11:05:03.106759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:50:25.499608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998940+0000","last_change":"2024-03-16T11:04:00.290522+0000","last_active":"2024-03-16T11:05:31.998940+0000","last_peered":"2024-03-16T11:05:31.998940+0000","last_clean":"2024-03-16T11:05:31.998940+0000","last_became_active":"2024-03-16T11:04:00.290175+0000","last_became_peered":"2024-03-16T11:04:00.290175+0000","last_unstale":"2024-03-16T11:05:31.998940+0000","last_undegraded":"2024-03-16T11:05:31.998940+0000","last_fullsized":"2024-03-16T11:05:31.998940+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:34.113278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548452+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548452+0000","last_peered":"2024-03-16T11:05:03.548452+0000","last_clean":"2024-03-16T11:05:03.548452+0000","last_became_active":"2024-03-16T11:03:59.289367+0000","last_became_peered":"2024-03-16T11:03:59.289367+0000","last_unstale":"2024-03-16T11:05:03.548452+0000","last_undegraded":"2024-03-16T11:05:03.548452+0000","last_fullsized":"2024-03-16T11:05:03.548452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:42:48.627116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994427+0000","last_change":"2024-03-16T11:04:00.287685+0000","last_active":"2024-03-16T11:05:31.994427+0000","last_peered":"2024-03-16T11:05:31.994427+0000","last_clean":"2024-03-16T11:05:31.994427+0000","last_became_active":"2024-03-16T11:04:00.287367+0000","last_became_peered":"2024-03-16T11:04:00.287367+0000","last_unstale":"2024-03-16T11:05:31.994427+0000","last_undegraded":"2024-03-16T11:05:31.994427+0000","last_fullsized":"2024-03-16T11:05:31.994427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:25:52.741651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108923+0000","last_change":"2024-03-16T11:03:59.286293+0000","last_active":"2024-03-16T11:05:03.108923+0000","last_peered":"2024-03-16T11:05:03.108923+0000","last_clean":"2024-03-16T11:05:03.108923+0000","last_became_active":"2024-03-16T11:03:59.285817+0000","last_became_peered":"2024-03-16T11:03:59.285817+0000","last_unstale":"2024-03-16T11:05:03.108923+0000","last_undegraded":"2024-03-16T11:05:03.108923+0000","last_fullsized":"2024-03-16T11:05:03.108923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:38:12.093762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008486+0000","last_change":"2024-03-16T11:04:00.299391+0000","last_active":"2024-03-16T11:05:32.008486+0000","last_peered":"2024-03-16T11:05:32.008486+0000","last_clean":"2024-03-16T11:05:32.008486+0000","last_became_active":"2024-03-16T11:04:00.298931+0000","last_became_peered":"2024-03-16T11:04:00.298931+0000","last_unstale":"2024-03-16T11:05:32.008486+0000","last_undegraded":"2024-03-16T11:05:32.008486+0000","last_fullsized":"2024-03-16T11:05:32.008486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:12:11.900154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548534+0000","last_change":"2024-03-16T11:03:59.292319+0000","last_active":"2024-03-16T11:05:03.548534+0000","last_peered":"2024-03-16T11:05:03.548534+0000","last_clean":"2024-03-16T11:05:03.548534+0000","last_became_active":"2024-03-16T11:03:59.291953+0000","last_became_peered":"2024-03-16T11:03:59.291953+0000","last_unstale":"2024-03-16T11:05:03.548534+0000","last_undegraded":"2024-03-16T11:05:03.548534+0000","last_fullsized":"2024-03-16T11:05:03.548534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:41:04.615726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008131+0000","last_change":"2024-03-16T11:04:00.296544+0000","last_active":"2024-03-16T11:05:32.008131+0000","last_peered":"2024-03-16T11:05:32.008131+0000","last_clean":"2024-03-16T11:05:32.008131+0000","last_became_active":"2024-03-16T11:04:00.296078+0000","last_became_peered":"2024-03-16T11:04:00.296078+0000","last_unstale":"2024-03-16T11:05:32.008131+0000","last_undegraded":"2024-03-16T11:05:32.008131+0000","last_fullsized":"2024-03-16T11:05:32.008131+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:56:22.846734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103250+0000","last_change":"2024-03-16T11:03:59.287172+0000","last_active":"2024-03-16T11:05:03.103250+0000","last_peered":"2024-03-16T11:05:03.103250+0000","last_clean":"2024-03-16T11:05:03.103250+0000","last_became_active":"2024-03-16T11:03:59.286785+0000","last_became_peered":"2024-03-16T11:03:59.286785+0000","last_unstale":"2024-03-16T11:05:03.103250+0000","last_undegraded":"2024-03-16T11:05:03.103250+0000","last_fullsized":"2024-03-16T11:05:03.103250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:17:55.584618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990540+0000","last_change":"2024-03-16T11:05:03.111534+0000","last_active":"2024-03-16T11:05:31.990540+0000","last_peered":"2024-03-16T11:05:31.990540+0000","last_clean":"2024-03-16T11:05:31.990540+0000","last_became_active":"2024-03-16T11:05:03.107517+0000","last_became_peered":"2024-03-16T11:05:03.107517+0000","last_unstale":"2024-03-16T11:05:31.990540+0000","last_undegraded":"2024-03-16T11:05:31.990540+0000","last_fullsized":"2024-03-16T11:05:31.990540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:07:51.828833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103117+0000","last_change":"2024-03-16T11:03:59.288822+0000","last_active":"2024-03-16T11:05:03.103117+0000","last_peered":"2024-03-16T11:05:03.103117+0000","last_clean":"2024-03-16T11:05:03.103117+0000","last_became_active":"2024-03-16T11:03:59.288516+0000","last_became_peered":"2024-03-16T11:03:59.288516+0000","last_unstale":"2024-03-16T11:05:03.103117+0000","last_undegraded":"2024-03-16T11:05:03.103117+0000","last_fullsized":"2024-03-16T11:05:03.103117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:32:38.862476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003380+0000","last_change":"2024-03-16T11:04:00.296117+0000","last_active":"2024-03-16T11:05:32.003380+0000","last_peered":"2024-03-16T11:05:32.003380+0000","last_clean":"2024-03-16T11:05:32.003380+0000","last_became_active":"2024-03-16T11:04:00.295664+0000","last_became_peered":"2024-03-16T11:04:00.295664+0000","last_unstale":"2024-03-16T11:05:32.003380+0000","last_undegraded":"2024-03-16T11:05:32.003380+0000","last_fullsized":"2024-03-16T11:05:32.003380+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.448616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999611+0000","last_change":"2024-03-16T11:04:00.295035+0000","last_active":"2024-03-16T11:05:31.999611+0000","last_peered":"2024-03-16T11:05:31.999611+0000","last_clean":"2024-03-16T11:05:31.999611+0000","last_became_active":"2024-03-16T11:04:00.294689+0000","last_became_peered":"2024-03-16T11:04:00.294689+0000","last_unstale":"2024-03-16T11:05:31.999611+0000","last_undegraded":"2024-03-16T11:05:31.999611+0000","last_fullsized":"2024-03-16T11:05:31.999611+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:55.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998519+0000","last_change":"2024-03-16T11:04:00.292831+0000","last_active":"2024-03-16T11:05:31.998519+0000","last_peered":"2024-03-16T11:05:31.998519+0000","last_clean":"2024-03-16T11:05:31.998519+0000","last_became_active":"2024-03-16T11:04:00.292387+0000","last_became_peered":"2024-03-16T11:04:00.292387+0000","last_unstale":"2024-03-16T11:05:31.998519+0000","last_undegraded":"2024-03-16T11:05:31.998519+0000","last_fullsized":"2024-03-16T11:05:31.998519+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:14:53.273535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.970952+0000","last_change":"2024-03-16T11:04:00.291267+0000","last_active":"2024-03-16T11:05:31.970952+0000","last_peered":"2024-03-16T11:05:31.970952+0000","last_clean":"2024-03-16T11:05:31.970952+0000","last_became_active":"2024-03-16T11:04:00.290911+0000","last_became_peered":"2024-03-16T11:04:00.290911+0000","last_unstale":"2024-03-16T11:05:31.970952+0000","last_undegraded":"2024-03-16T11:05:31.970952+0000","last_fullsized":"2024-03-16T11:05:31.970952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:35.285167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999212+0000","last_change":"2024-03-16T11:04:00.300470+0000","last_active":"2024-03-16T11:05:31.999212+0000","last_peered":"2024-03-16T11:05:31.999212+0000","last_clean":"2024-03-16T11:05:31.999212+0000","last_became_active":"2024-03-16T11:04:00.300107+0000","last_became_peered":"2024-03-16T11:04:00.300107+0000","last_unstale":"2024-03-16T11:05:31.999212+0000","last_undegraded":"2024-03-16T11:05:31.999212+0000","last_fullsized":"2024-03-16T11:05:31.999212+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:18.718749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.001171+0000","last_change":"2024-03-16T11:04:00.300857+0000","last_active":"2024-03-16T11:05:32.001171+0000","last_peered":"2024-03-16T11:05:32.001171+0000","last_clean":"2024-03-16T11:05:32.001171+0000","last_became_active":"2024-03-16T11:04:00.300551+0000","last_became_peered":"2024-03-16T11:04:00.300551+0000","last_unstale":"2024-03-16T11:05:32.001171+0000","last_undegraded":"2024-03-16T11:05:32.001171+0000","last_fullsized":"2024-03-16T11:05:32.001171+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:24.362561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.996929+0000","last_change":"2024-03-16T11:05:03.110781+0000","last_active":"2024-03-16T11:05:31.996929+0000","last_peered":"2024-03-16T11:05:31.996929+0000","last_clean":"2024-03-16T11:05:31.996929+0000","last_became_active":"2024-03-16T11:05:03.110375+0000","last_became_peered":"2024-03-16T11:05:03.110375+0000","last_unstale":"2024-03-16T11:05:31.996929+0000","last_undegraded":"2024-03-16T11:05:31.996929+0000","last_fullsized":"2024-03-16T11:05:31.996929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:28.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998128+0000","last_change":"2024-03-16T11:04:00.294943+0000","last_active":"2024-03-16T11:05:31.998128+0000","last_peered":"2024-03-16T11:05:31.998128+0000","last_clean":"2024-03-16T11:05:31.998128+0000","last_became_active":"2024-03-16T11:04:00.294637+0000","last_became_peered":"2024-03-16T11:04:00.294637+0000","last_unstale":"2024-03-16T11:05:31.998128+0000","last_undegraded":"2024-03-16T11:05:31.998128+0000","last_fullsized":"2024-03-16T11:05:31.998128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:03:09.716753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990054+0000","last_change":"2024-03-16T11:04:00.300737+0000","last_active":"2024-03-16T11:05:31.990054+0000","last_peered":"2024-03-16T11:05:31.990054+0000","last_clean":"2024-03-16T11:05:31.990054+0000","last_became_active":"2024-03-16T11:04:00.300410+0000","last_became_peered":"2024-03-16T11:04:00.300410+0000","last_unstale":"2024-03-16T11:05:31.990054+0000","last_undegraded":"2024-03-16T11:05:31.990054+0000","last_fullsized":"2024-03-16T11:05:31.990054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:18.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.971404+0000","last_change":"2024-03-16T11:04:00.292958+0000","last_active":"2024-03-16T11:05:31.971404+0000","last_peered":"2024-03-16T11:05:31.971404+0000","last_clean":"2024-03-16T11:05:31.971404+0000","last_became_active":"2024-03-16T11:04:00.292599+0000","last_became_peered":"2024-03-16T11:04:00.292599+0000","last_unstale":"2024-03-16T11:05:31.971404+0000","last_undegraded":"2024-03-16T11:05:31.971404+0000","last_fullsized":"2024-03-16T11:05:31.971404+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:46.370597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994683+0000","last_change":"2024-03-16T11:04:00.291381+0000","last_active":"2024-03-16T11:05:31.994683+0000","last_peered":"2024-03-16T11:05:31.994683+0000","last_clean":"2024-03-16T11:05:31.994683+0000","last_became_active":"2024-03-16T11:04:00.291046+0000","last_became_peered":"2024-03-16T11:04:00.291046+0000","last_unstale":"2024-03-16T11:05:31.994683+0000","last_undegraded":"2024-03-16T11:05:31.994683+0000","last_fullsized":"2024-03-16T11:05:31.994683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:08.185733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008021+0000","last_change":"2024-03-16T11:04:00.301381+0000","last_active":"2024-03-16T11:05:32.008021+0000","last_peered":"2024-03-16T11:05:32.008021+0000","last_clean":"2024-03-16T11:05:32.008021+0000","last_became_active":"2024-03-16T11:04:00.300990+0000","last_became_peered":"2024-03-16T11:04:00.300990+0000","last_unstale":"2024-03-16T11:05:32.008021+0000","last_undegraded":"2024-03-16T11:05:32.008021+0000","last_fullsized":"2024-03-16T11:05:32.008021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:07:43.775066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999944+0000","last_change":"2024-03-16T11:04:00.298220+0000","last_active":"2024-03-16T11:05:31.999944+0000","last_peered":"2024-03-16T11:05:31.999944+0000","last_clean":"2024-03-16T11:05:31.999944+0000","last_became_active":"2024-03-16T11:04:00.297776+0000","last_became_peered":"2024-03-16T11:04:00.297776+0000","last_unstale":"2024-03-16T11:05:31.999944+0000","last_undegraded":"2024-03-16T11:05:31.999944+0000","last_fullsized":"2024-03-16T11:05:31.999944+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:40.387309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999183+0000","last_change":"2024-03-16T11:04:00.292988+0000","last_active":"2024-03-16T11:05:31.999183+0000","last_peered":"2024-03-16T11:05:31.999183+0000","last_clean":"2024-03-16T11:05:31.999183+0000","last_became_active":"2024-03-16T11:04:00.292655+0000","last_became_peered":"2024-03-16T11:04:00.292655+0000","last_unstale":"2024-03-16T11:05:31.999183+0000","last_undegraded":"2024-03-16T11:05:31.999183+0000","last_fullsized":"2024-03-16T11:05:31.999183+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:33:43.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71813,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":702,"num_read_kb":424,"num_write":170,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":199303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285604,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":471093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547242,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1683456,"data_stored":1128327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743471,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":471307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005107,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":471205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1593344,"data_stored":1063298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430270,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":470813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":529650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":471967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":531000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":471007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":472158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1601536,"data_stored":1063217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:05:57.878 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-16T11:05:57.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 -- 172.21.15.78:0/50102920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441018e0 msgr2=0x7f4344101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 --2- 172.21.15.78:0/50102920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441018e0 0x7f4344101d60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f432c009900 tx=0x7f432c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 -- 172.21.15.78:0/50102920 shutdown_connections 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 --2- 172.21.15.78:0/50102920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43441022a0 0x7f4344108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 --2- 172.21.15.78:0/50102920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441018e0 0x7f4344101d60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 --2- 172.21.15.78:0/50102920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43441006f0 0x7f4344100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 -- 172.21.15.78:0/50102920 >> 172.21.15.78:0/50102920 conn(0x7f43440fbc60 msgr2=0x7f43440fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 -- 172.21.15.78:0/50102920 shutdown_connections 2024-03-16T11:05:58.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.015+0000 7f4348a30640 1 -- 172.21.15.78:0/50102920 wait complete. 2024-03-16T11:05:58.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 Processor -- start 2024-03-16T11:05:58.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 -- start start 2024-03-16T11:05:58.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441006f0 0x7f434406aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43441018e0 0x7f434419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43441022a0 0x7f434419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f434410c220 con 0x7f43441006f0 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f434410c0a0 con 0x7f43441022a0 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4348a30640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f434410c3a0 con 0x7f43441018e0 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441006f0 0x7f434406aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441006f0 0x7f434406aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54812/0 (socket says 172.21.15.78:54812) 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 -- 172.21.15.78:0/2596445335 learned_addr learned my addr 172.21.15.78:0/2596445335 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f433bfff640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43441018e0 0x7f434419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f4342d9d640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43441022a0 0x7f434419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 -- 172.21.15.78:0/2596445335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43441018e0 msgr2=0x7f434419c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43441018e0 0x7f434419c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 -- 172.21.15.78:0/2596445335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43441022a0 msgr2=0x7f434419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43441022a0 0x7f434419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.016+0000 7f434259c640 1 -- 172.21.15.78:0/2596445335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434419f680 con 0x7f43441006f0 2024-03-16T11:05:58.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.017+0000 7f434259c640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441006f0 0x7f434406aa20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f433400e9c0 tx=0x7f433400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.017+0000 7f433b7fe640 1 -- 172.21.15.78:0/2596445335 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433400cde0 con 0x7f43441006f0 2024-03-16T11:05:58.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.017+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434419f910 con 0x7f43441006f0 2024-03-16T11:05:58.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.017+0000 7f433b7fe640 1 -- 172.21.15.78:0/2596445335 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f43340108e0 con 0x7f43441006f0 2024-03-16T11:05:58.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.017+0000 7f433b7fe640 1 -- 172.21.15.78:0/2596445335 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43340188e0 con 0x7f43441006f0 2024-03-16T11:05:58.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.017+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434419fe30 con 0x7f43441006f0 2024-03-16T11:05:58.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.018+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4344107a90 con 0x7f43441006f0 2024-03-16T11:05:58.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.019+0000 7f433b7fe640 1 -- 172.21.15.78:0/2596445335 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f433401f020 con 0x7f43441006f0 2024-03-16T11:05:58.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.020+0000 7f433b7fe640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f43300a2890 0x7f43300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.020+0000 7f433bfff640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f43300a2890 0x7f43300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.020+0000 7f433b7fe640 1 -- 172.21.15.78:0/2596445335 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f43340cedb0 con 0x7f43441006f0 2024-03-16T11:05:58.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.020+0000 7f433bfff640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f43300a2890 0x7f43300a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f432c009230 tx=0x7f432c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.024+0000 7f433b7fe640 1 -- 172.21.15.78:0/2596445335 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f433409bcf0 con 0x7f43441006f0 2024-03-16T11:05:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.196+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 --> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4344061c20 con 0x7f43300a2890 2024-03-16T11:05:58.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.203+0000 7f433b7fe640 1 -- 172.21.15.78:0/2596445335 <== mgr.14152 v2:172.21.15.78:6800/4254906951 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258839 (secure 0 0 0) 0x7f4344061c20 con 0x7f43300a2890 2024-03-16T11:05:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-16T11:05:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f43300a2890 msgr2=0x7f43300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f43300a2890 0x7f43300a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f432c009230 tx=0x7f432c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441006f0 msgr2=0x7f434406aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441006f0 0x7f434406aa20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f433400e9c0 tx=0x7f433400ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 shutdown_connections 2024-03-16T11:05:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43441022a0 0x7f434419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f43300a2890 0x7f43300a4d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43441018e0 0x7f434419c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 --2- 172.21.15.78:0/2596445335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43441006f0 0x7f434406aa20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 >> 172.21.15.78:0/2596445335 conn(0x7f43440fbc60 msgr2=0x7f4344104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.206+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 shutdown_connections 2024-03-16T11:05:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.207+0000 7f4348a30640 1 -- 172.21.15.78:0/2596445335 wait complete. 2024-03-16T11:05:58.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:58 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3011666219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:58.218 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-03-16T11:05:57.993997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662415,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":828,"num_read_kb":533,"num_write":359,"num_write_kb":3067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383616,"kb_used_data":13664,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558208,"statfs":{"total":1151940427776,"available":1151547604992,"internally_reserved":0,"allocated":13991936,"data_stored":7630192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8510,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":39,"num_write":15,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002856"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008007+0000","last_change":"2024-03-16T11:04:00.295528+0000","last_active":"2024-03-16T11:05:32.008007+0000","last_peered":"2024-03-16T11:05:32.008007+0000","last_clean":"2024-03-16T11:05:32.008007+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:32.008007+0000","last_undegraded":"2024-03-16T11:05:32.008007+0000","last_fullsized":"2024-03-16T11:05:32.008007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:22.852488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994520+0000","last_change":"2024-03-16T11:04:00.289043+0000","last_active":"2024-03-16T11:05:31.994520+0000","last_peered":"2024-03-16T11:05:31.994520+0000","last_clean":"2024-03-16T11:05:31.994520+0000","last_became_active":"2024-03-16T11:04:00.288742+0000","last_became_peered":"2024-03-16T11:04:00.288742+0000","last_unstale":"2024-03-16T11:05:31.994520+0000","last_undegraded":"2024-03-16T11:05:31.994520+0000","last_fullsized":"2024-03-16T11:05:31.994520+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:07:56.710499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007950+0000","last_change":"2024-03-16T11:04:00.293443+0000","last_active":"2024-03-16T11:05:32.007950+0000","last_peered":"2024-03-16T11:05:32.007950+0000","last_clean":"2024-03-16T11:05:32.007950+0000","last_became_active":"2024-03-16T11:04:00.293117+0000","last_became_peered":"2024-03-16T11:04:00.293117+0000","last_unstale":"2024-03-16T11:05:32.007950+0000","last_undegraded":"2024-03-16T11:05:32.007950+0000","last_fullsized":"2024-03-16T11:05:32.007950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:40:49.728721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998576+0000","last_change":"2024-03-16T11:04:00.292105+0000","last_active":"2024-03-16T11:05:31.998576+0000","last_peered":"2024-03-16T11:05:31.998576+0000","last_clean":"2024-03-16T11:05:31.998576+0000","last_became_active":"2024-03-16T11:04:00.291755+0000","last_became_peered":"2024-03-16T11:04:00.291755+0000","last_unstale":"2024-03-16T11:05:31.998576+0000","last_undegraded":"2024-03-16T11:05:31.998576+0000","last_fullsized":"2024-03-16T11:05:31.998576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:48:43.734854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.997439+0000","last_change":"2024-03-16T11:04:00.282168+0000","last_active":"2024-03-16T11:05:31.997439+0000","last_peered":"2024-03-16T11:05:31.997439+0000","last_clean":"2024-03-16T11:05:31.997439+0000","last_became_active":"2024-03-16T11:04:00.281753+0000","last_became_peered":"2024-03-16T11:04:00.281753+0000","last_unstale":"2024-03-16T11:05:31.997439+0000","last_undegraded":"2024-03-16T11:05:31.997439+0000","last_fullsized":"2024-03-16T11:05:31.997439+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:13.223955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002045+0000","last_change":"2024-03-16T11:05:03.111462+0000","last_active":"2024-03-16T11:05:32.002045+0000","last_peered":"2024-03-16T11:05:32.002045+0000","last_clean":"2024-03-16T11:05:32.002045+0000","last_became_active":"2024-03-16T11:05:03.111087+0000","last_became_peered":"2024-03-16T11:05:03.111087+0000","last_unstale":"2024-03-16T11:05:32.002045+0000","last_undegraded":"2024-03-16T11:05:32.002045+0000","last_fullsized":"2024-03-16T11:05:32.002045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:09.339732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999489+0000","last_change":"2024-03-16T11:04:00.292446+0000","last_active":"2024-03-16T11:05:31.999489+0000","last_peered":"2024-03-16T11:05:31.999489+0000","last_clean":"2024-03-16T11:05:31.999489+0000","last_became_active":"2024-03-16T11:04:00.292079+0000","last_became_peered":"2024-03-16T11:04:00.292079+0000","last_unstale":"2024-03-16T11:05:31.999489+0000","last_undegraded":"2024-03-16T11:05:31.999489+0000","last_fullsized":"2024-03-16T11:05:31.999489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.325945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003952+0000","last_change":"2024-03-16T11:04:00.294644+0000","last_active":"2024-03-16T11:05:32.003952+0000","last_peered":"2024-03-16T11:05:32.003952+0000","last_clean":"2024-03-16T11:05:32.003952+0000","last_became_active":"2024-03-16T11:04:00.294283+0000","last_became_peered":"2024-03-16T11:04:00.294283+0000","last_unstale":"2024-03-16T11:05:32.003952+0000","last_undegraded":"2024-03-16T11:05:32.003952+0000","last_fullsized":"2024-03-16T11:05:32.003952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:43.125823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007001+0000","last_change":"2024-03-16T11:04:00.302512+0000","last_active":"2024-03-16T11:05:32.007001+0000","last_peered":"2024-03-16T11:05:32.007001+0000","last_clean":"2024-03-16T11:05:32.007001+0000","last_became_active":"2024-03-16T11:04:00.301891+0000","last_became_peered":"2024-03-16T11:04:00.301891+0000","last_unstale":"2024-03-16T11:05:32.007001+0000","last_undegraded":"2024-03-16T11:05:32.007001+0000","last_fullsized":"2024-03-16T11:05:32.007001+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:15.112295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003148+0000","last_change":"2024-03-16T11:04:00.293327+0000","last_active":"2024-03-16T11:05:32.003148+0000","last_peered":"2024-03-16T11:05:32.003148+0000","last_clean":"2024-03-16T11:05:32.003148+0000","last_became_active":"2024-03-16T11:04:00.292971+0000","last_became_peered":"2024-03-16T11:04:00.292971+0000","last_unstale":"2024-03-16T11:05:32.003148+0000","last_undegraded":"2024-03-16T11:05:32.003148+0000","last_fullsized":"2024-03-16T11:05:32.003148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:48:48.565861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003814+0000","last_change":"2024-03-16T11:04:00.298956+0000","last_active":"2024-03-16T11:05:32.003814+0000","last_peered":"2024-03-16T11:05:32.003814+0000","last_clean":"2024-03-16T11:05:32.003814+0000","last_became_active":"2024-03-16T11:04:00.298508+0000","last_became_peered":"2024-03-16T11:04:00.298508+0000","last_unstale":"2024-03-16T11:05:32.003814+0000","last_undegraded":"2024-03-16T11:05:32.003814+0000","last_fullsized":"2024-03-16T11:05:32.003814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:25:46.576514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002545+0000","last_change":"2024-03-16T11:04:00.294453+0000","last_active":"2024-03-16T11:05:32.002545+0000","last_peered":"2024-03-16T11:05:32.002545+0000","last_clean":"2024-03-16T11:05:32.002545+0000","last_became_active":"2024-03-16T11:04:00.294084+0000","last_became_peered":"2024-03-16T11:04:00.294084+0000","last_unstale":"2024-03-16T11:05:32.002545+0000","last_undegraded":"2024-03-16T11:05:32.002545+0000","last_fullsized":"2024-03-16T11:05:32.002545+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:12.924835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003591+0000","last_change":"2024-03-16T11:04:00.293149+0000","last_active":"2024-03-16T11:05:32.003591+0000","last_peered":"2024-03-16T11:05:32.003591+0000","last_clean":"2024-03-16T11:05:32.003591+0000","last_became_active":"2024-03-16T11:04:00.292839+0000","last_became_peered":"2024-03-16T11:04:00.292839+0000","last_unstale":"2024-03-16T11:05:32.003591+0000","last_undegraded":"2024-03-16T11:05:32.003591+0000","last_fullsized":"2024-03-16T11:05:32.003591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:34:38.446918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.993502+0000","last_change":"2024-03-16T11:04:00.285748+0000","last_active":"2024-03-16T11:05:31.993502+0000","last_peered":"2024-03-16T11:05:31.993502+0000","last_clean":"2024-03-16T11:05:31.993502+0000","last_became_active":"2024-03-16T11:04:00.285394+0000","last_became_peered":"2024-03-16T11:04:00.285394+0000","last_unstale":"2024-03-16T11:05:31.993502+0000","last_undegraded":"2024-03-16T11:05:31.993502+0000","last_fullsized":"2024-03-16T11:05:31.993502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:02.113271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999076+0000","last_change":"2024-03-16T11:04:00.301783+0000","last_active":"2024-03-16T11:05:31.999076+0000","last_peered":"2024-03-16T11:05:31.999076+0000","last_clean":"2024-03-16T11:05:31.999076+0000","last_became_active":"2024-03-16T11:04:00.301313+0000","last_became_peered":"2024-03-16T11:04:00.301313+0000","last_unstale":"2024-03-16T11:05:31.999076+0000","last_undegraded":"2024-03-16T11:05:31.999076+0000","last_fullsized":"2024-03-16T11:05:31.999076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:19:32.243602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007419+0000","last_change":"2024-03-16T11:04:00.301375+0000","last_active":"2024-03-16T11:05:32.007419+0000","last_peered":"2024-03-16T11:05:32.007419+0000","last_clean":"2024-03-16T11:05:32.007419+0000","last_became_active":"2024-03-16T11:04:00.301013+0000","last_became_peered":"2024-03-16T11:04:00.301013+0000","last_unstale":"2024-03-16T11:05:32.007419+0000","last_undegraded":"2024-03-16T11:05:32.007419+0000","last_fullsized":"2024-03-16T11:05:32.007419+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:21:52.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008074+0000","last_change":"2024-03-16T11:04:00.296287+0000","last_active":"2024-03-16T11:05:32.008074+0000","last_peered":"2024-03-16T11:05:32.008074+0000","last_clean":"2024-03-16T11:05:32.008074+0000","last_became_active":"2024-03-16T11:04:00.295972+0000","last_became_peered":"2024-03-16T11:04:00.295972+0000","last_unstale":"2024-03-16T11:05:32.008074+0000","last_undegraded":"2024-03-16T11:05:32.008074+0000","last_fullsized":"2024-03-16T11:05:32.008074+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:14.586052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006625+0000","last_change":"2024-03-16T11:04:00.295551+0000","last_active":"2024-03-16T11:05:32.006625+0000","last_peered":"2024-03-16T11:05:32.006625+0000","last_clean":"2024-03-16T11:05:32.006625+0000","last_became_active":"2024-03-16T11:04:00.295185+0000","last_became_peered":"2024-03-16T11:04:00.295185+0000","last_unstale":"2024-03-16T11:05:32.006625+0000","last_undegraded":"2024-03-16T11:05:32.006625+0000","last_fullsized":"2024-03-16T11:05:32.006625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:53.495224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.971033+0000","last_change":"2024-03-16T11:04:00.290455+0000","last_active":"2024-03-16T11:05:31.971033+0000","last_peered":"2024-03-16T11:05:31.971033+0000","last_clean":"2024-03-16T11:05:31.971033+0000","last_became_active":"2024-03-16T11:04:00.290057+0000","last_became_peered":"2024-03-16T11:04:00.290057+0000","last_unstale":"2024-03-16T11:05:31.971033+0000","last_undegraded":"2024-03-16T11:05:31.971033+0000","last_fullsized":"2024-03-16T11:05:31.971033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:57:43.677050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102676+0000","last_change":"2024-03-16T11:03:59.285059+0000","last_active":"2024-03-16T11:05:03.102676+0000","last_peered":"2024-03-16T11:05:03.102676+0000","last_clean":"2024-03-16T11:05:03.102676+0000","last_became_active":"2024-03-16T11:03:59.284745+0000","last_became_peered":"2024-03-16T11:03:59.284745+0000","last_unstale":"2024-03-16T11:05:03.102676+0000","last_undegraded":"2024-03-16T11:05:03.102676+0000","last_fullsized":"2024-03-16T11:05:03.102676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:12.516958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990186+0000","last_change":"2024-03-16T11:04:00.292933+0000","last_active":"2024-03-16T11:05:31.990186+0000","last_peered":"2024-03-16T11:05:31.990186+0000","last_clean":"2024-03-16T11:05:31.990186+0000","last_became_active":"2024-03-16T11:04:00.292637+0000","last_became_peered":"2024-03-16T11:04:00.292637+0000","last_unstale":"2024-03-16T11:05:31.990186+0000","last_undegraded":"2024-03-16T11:05:31.990186+0000","last_fullsized":"2024-03-16T11:05:31.990186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:22.231735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.107896+0000","last_change":"2024-03-16T11:03:59.285267+0000","last_active":"2024-03-16T11:05:03.107896+0000","last_peered":"2024-03-16T11:05:03.107896+0000","last_clean":"2024-03-16T11:05:03.107896+0000","last_became_active":"2024-03-16T11:03:59.284890+0000","last_became_peered":"2024-03-16T11:03:59.284890+0000","last_unstale":"2024-03-16T11:05:03.107896+0000","last_undegraded":"2024-03-16T11:05:03.107896+0000","last_fullsized":"2024-03-16T11:05:03.107896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:53.625612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008052+0000","last_change":"2024-03-16T11:04:00.292660+0000","last_active":"2024-03-16T11:05:32.008052+0000","last_peered":"2024-03-16T11:05:32.008052+0000","last_clean":"2024-03-16T11:05:32.008052+0000","last_became_active":"2024-03-16T11:04:00.292366+0000","last_became_peered":"2024-03-16T11:04:00.292366+0000","last_unstale":"2024-03-16T11:05:32.008052+0000","last_undegraded":"2024-03-16T11:05:32.008052+0000","last_fullsized":"2024-03-16T11:05:32.008052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:36:44.331456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109296+0000","last_change":"2024-03-16T11:03:59.286370+0000","last_active":"2024-03-16T11:05:03.109296+0000","last_peered":"2024-03-16T11:05:03.109296+0000","last_clean":"2024-03-16T11:05:03.109296+0000","last_became_active":"2024-03-16T11:03:59.286065+0000","last_became_peered":"2024-03-16T11:03:59.286065+0000","last_unstale":"2024-03-16T11:05:03.109296+0000","last_undegraded":"2024-03-16T11:05:03.109296+0000","last_fullsized":"2024-03-16T11:05:03.109296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:20.126848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007484+0000","last_change":"2024-03-16T11:04:00.290864+0000","last_active":"2024-03-16T11:05:32.007484+0000","last_peered":"2024-03-16T11:05:32.007484+0000","last_clean":"2024-03-16T11:05:32.007484+0000","last_became_active":"2024-03-16T11:04:00.290537+0000","last_became_peered":"2024-03-16T11:04:00.290537+0000","last_unstale":"2024-03-16T11:05:32.007484+0000","last_undegraded":"2024-03-16T11:05:32.007484+0000","last_fullsized":"2024-03-16T11:05:32.007484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:52.991767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109155+0000","last_change":"2024-03-16T11:03:59.281663+0000","last_active":"2024-03-16T11:05:03.109155+0000","last_peered":"2024-03-16T11:05:03.109155+0000","last_clean":"2024-03-16T11:05:03.109155+0000","last_became_active":"2024-03-16T11:03:59.281328+0000","last_became_peered":"2024-03-16T11:03:59.281328+0000","last_unstale":"2024-03-16T11:05:03.109155+0000","last_undegraded":"2024-03-16T11:05:03.109155+0000","last_fullsized":"2024-03-16T11:05:03.109155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:44.511109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003970+0000","last_change":"2024-03-16T11:05:03.106586+0000","last_active":"2024-03-16T11:05:32.003970+0000","last_peered":"2024-03-16T11:05:32.003970+0000","last_clean":"2024-03-16T11:05:32.003970+0000","last_became_active":"2024-03-16T11:05:03.106233+0000","last_became_peered":"2024-03-16T11:05:03.106233+0000","last_unstale":"2024-03-16T11:05:32.003970+0000","last_undegraded":"2024-03-16T11:05:32.003970+0000","last_fullsized":"2024-03-16T11:05:32.003970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:19:30.480318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":142,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:54.588491+0000","last_change":"2024-03-16T11:03:59.286300+0000","last_active":"2024-03-16T11:05:54.588491+0000","last_peered":"2024-03-16T11:05:54.588491+0000","last_clean":"2024-03-16T11:05:54.588491+0000","last_became_active":"2024-03-16T11:03:59.285764+0000","last_became_peered":"2024-03-16T11:03:59.285764+0000","last_unstale":"2024-03-16T11:05:54.588491+0000","last_undegraded":"2024-03-16T11:05:54.588491+0000","last_fullsized":"2024-03-16T11:05:54.588491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:49.084245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.984380+0000","last_change":"2024-03-16T11:04:00.289152+0000","last_active":"2024-03-16T11:05:31.984380+0000","last_peered":"2024-03-16T11:05:31.984380+0000","last_clean":"2024-03-16T11:05:31.984380+0000","last_became_active":"2024-03-16T11:04:00.288837+0000","last_became_peered":"2024-03-16T11:04:00.288837+0000","last_unstale":"2024-03-16T11:05:31.984380+0000","last_undegraded":"2024-03-16T11:05:31.984380+0000","last_fullsized":"2024-03-16T11:05:31.984380+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:45:01.368129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106073+0000","last_change":"2024-03-16T11:03:59.281505+0000","last_active":"2024-03-16T11:05:03.106073+0000","last_peered":"2024-03-16T11:05:03.106073+0000","last_clean":"2024-03-16T11:05:03.106073+0000","last_became_active":"2024-03-16T11:03:59.281175+0000","last_became_peered":"2024-03-16T11:03:59.281175+0000","last_unstale":"2024-03-16T11:05:03.106073+0000","last_undegraded":"2024-03-16T11:05:03.106073+0000","last_fullsized":"2024-03-16T11:05:03.106073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:12.463914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.988807+0000","last_change":"2024-03-16T11:04:00.297769+0000","last_active":"2024-03-16T11:05:31.988807+0000","last_peered":"2024-03-16T11:05:31.988807+0000","last_clean":"2024-03-16T11:05:31.988807+0000","last_became_active":"2024-03-16T11:04:00.297061+0000","last_became_peered":"2024-03-16T11:04:00.297061+0000","last_unstale":"2024-03-16T11:05:31.988807+0000","last_undegraded":"2024-03-16T11:05:31.988807+0000","last_fullsized":"2024-03-16T11:05:31.988807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:42.594315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103214+0000","last_change":"2024-03-16T11:03:59.290260+0000","last_active":"2024-03-16T11:05:03.103214+0000","last_peered":"2024-03-16T11:05:03.103214+0000","last_clean":"2024-03-16T11:05:03.103214+0000","last_became_active":"2024-03-16T11:03:59.289843+0000","last_became_peered":"2024-03-16T11:03:59.289843+0000","last_unstale":"2024-03-16T11:05:03.103214+0000","last_undegraded":"2024-03-16T11:05:03.103214+0000","last_fullsized":"2024-03-16T11:05:03.103214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:22.483488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.997150+0000","last_change":"2024-03-16T11:04:00.287873+0000","last_active":"2024-03-16T11:05:31.997150+0000","last_peered":"2024-03-16T11:05:31.997150+0000","last_clean":"2024-03-16T11:05:31.997150+0000","last_became_active":"2024-03-16T11:04:00.287002+0000","last_became_peered":"2024-03-16T11:04:00.287002+0000","last_unstale":"2024-03-16T11:05:31.997150+0000","last_undegraded":"2024-03-16T11:05:31.997150+0000","last_fullsized":"2024-03-16T11:05:31.997150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:57.851726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102815+0000","last_change":"2024-03-16T11:03:59.284745+0000","last_active":"2024-03-16T11:05:03.102815+0000","last_peered":"2024-03-16T11:05:03.102815+0000","last_clean":"2024-03-16T11:05:03.102815+0000","last_became_active":"2024-03-16T11:03:59.284325+0000","last_became_peered":"2024-03-16T11:03:59.284325+0000","last_unstale":"2024-03-16T11:05:03.102815+0000","last_undegraded":"2024-03-16T11:05:03.102815+0000","last_fullsized":"2024-03-16T11:05:03.102815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.537787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002828+0000","last_change":"2024-03-16T11:04:00.290698+0000","last_active":"2024-03-16T11:05:32.002828+0000","last_peered":"2024-03-16T11:05:32.002828+0000","last_clean":"2024-03-16T11:05:32.002828+0000","last_became_active":"2024-03-16T11:04:00.290348+0000","last_became_peered":"2024-03-16T11:04:00.290348+0000","last_unstale":"2024-03-16T11:05:32.002828+0000","last_undegraded":"2024-03-16T11:05:32.002828+0000","last_fullsized":"2024-03-16T11:05:32.002828+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:44.114305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":135,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.586690+0000","last_change":"2024-03-16T11:03:59.280705+0000","last_active":"2024-03-16T11:05:53.586690+0000","last_peered":"2024-03-16T11:05:53.586690+0000","last_clean":"2024-03-16T11:05:53.586690+0000","last_became_active":"2024-03-16T11:03:59.280372+0000","last_became_peered":"2024-03-16T11:03:59.280372+0000","last_unstale":"2024-03-16T11:05:53.586690+0000","last_undegraded":"2024-03-16T11:05:53.586690+0000","last_fullsized":"2024-03-16T11:05:53.586690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:54.187322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006386+0000","last_change":"2024-03-16T11:04:00.293867+0000","last_active":"2024-03-16T11:05:32.006386+0000","last_peered":"2024-03-16T11:05:32.006386+0000","last_clean":"2024-03-16T11:05:32.006386+0000","last_became_active":"2024-03-16T11:04:00.293330+0000","last_became_peered":"2024-03-16T11:04:00.293330+0000","last_unstale":"2024-03-16T11:05:32.006386+0000","last_undegraded":"2024-03-16T11:05:32.006386+0000","last_fullsized":"2024-03-16T11:05:32.006386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:54.798346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105547+0000","last_change":"2024-03-16T11:03:59.284202+0000","last_active":"2024-03-16T11:05:03.105547+0000","last_peered":"2024-03-16T11:05:03.105547+0000","last_clean":"2024-03-16T11:05:03.105547+0000","last_became_active":"2024-03-16T11:03:59.283656+0000","last_became_peered":"2024-03-16T11:03:59.283656+0000","last_unstale":"2024-03-16T11:05:03.105547+0000","last_undegraded":"2024-03-16T11:05:03.105547+0000","last_fullsized":"2024-03-16T11:05:03.105547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:44.760307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.996327+0000","last_change":"2024-03-16T11:05:03.111443+0000","last_active":"2024-03-16T11:05:31.996327+0000","last_peered":"2024-03-16T11:05:31.996327+0000","last_clean":"2024-03-16T11:05:31.996327+0000","last_became_active":"2024-03-16T11:05:03.111027+0000","last_became_peered":"2024-03-16T11:05:03.111027+0000","last_unstale":"2024-03-16T11:05:31.996327+0000","last_undegraded":"2024-03-16T11:05:31.996327+0000","last_fullsized":"2024-03-16T11:05:31.996327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:31.844283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103330+0000","last_change":"2024-03-16T11:03:59.288268+0000","last_active":"2024-03-16T11:05:03.103330+0000","last_peered":"2024-03-16T11:05:03.103330+0000","last_clean":"2024-03-16T11:05:03.103330+0000","last_became_active":"2024-03-16T11:03:59.287861+0000","last_became_peered":"2024-03-16T11:03:59.287861+0000","last_unstale":"2024-03-16T11:05:03.103330+0000","last_undegraded":"2024-03-16T11:05:03.103330+0000","last_fullsized":"2024-03-16T11:05:03.103330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:34:34.248408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.007936+0000","last_change":"2024-03-16T11:04:00.306490+0000","last_active":"2024-03-16T11:05:32.007936+0000","last_peered":"2024-03-16T11:05:32.007936+0000","last_clean":"2024-03-16T11:05:32.007936+0000","last_became_active":"2024-03-16T11:04:00.305874+0000","last_became_peered":"2024-03-16T11:04:00.305874+0000","last_unstale":"2024-03-16T11:05:32.007936+0000","last_undegraded":"2024-03-16T11:05:32.007936+0000","last_fullsized":"2024-03-16T11:05:32.007936+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":286,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:54.589113+0000","last_change":"2024-03-16T11:03:59.280714+0000","last_active":"2024-03-16T11:05:54.589113+0000","last_peered":"2024-03-16T11:05:54.589113+0000","last_clean":"2024-03-16T11:05:54.589113+0000","last_became_active":"2024-03-16T11:03:59.280398+0000","last_became_peered":"2024-03-16T11:03:59.280398+0000","last_unstale":"2024-03-16T11:05:54.589113+0000","last_undegraded":"2024-03-16T11:05:54.589113+0000","last_fullsized":"2024-03-16T11:05:54.589113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:34.256125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66294,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":234,"num_read_kb":179,"num_write":34,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999739+0000","last_change":"2024-03-16T11:04:00.285669+0000","last_active":"2024-03-16T11:05:31.999739+0000","last_peered":"2024-03-16T11:05:31.999739+0000","last_clean":"2024-03-16T11:05:31.999739+0000","last_became_active":"2024-03-16T11:04:00.285337+0000","last_became_peered":"2024-03-16T11:04:00.285337+0000","last_unstale":"2024-03-16T11:05:31.999739+0000","last_undegraded":"2024-03-16T11:05:31.999739+0000","last_fullsized":"2024-03-16T11:05:31.999739+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:36.925350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548506+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548506+0000","last_peered":"2024-03-16T11:05:03.548506+0000","last_clean":"2024-03-16T11:05:03.548506+0000","last_became_active":"2024-03-16T11:03:59.289357+0000","last_became_peered":"2024-03-16T11:03:59.289357+0000","last_unstale":"2024-03-16T11:05:03.548506+0000","last_undegraded":"2024-03-16T11:05:03.548506+0000","last_fullsized":"2024-03-16T11:05:03.548506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:26:35.860091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998250+0000","last_change":"2024-03-16T11:04:00.298488+0000","last_active":"2024-03-16T11:05:31.998250+0000","last_peered":"2024-03-16T11:05:31.998250+0000","last_clean":"2024-03-16T11:05:31.998250+0000","last_became_active":"2024-03-16T11:04:00.298202+0000","last_became_peered":"2024-03-16T11:04:00.298202+0000","last_unstale":"2024-03-16T11:05:31.998250+0000","last_undegraded":"2024-03-16T11:05:31.998250+0000","last_fullsized":"2024-03-16T11:05:31.998250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:48.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.102961+0000","last_change":"2024-03-16T11:03:59.287066+0000","last_active":"2024-03-16T11:05:03.102961+0000","last_peered":"2024-03-16T11:05:03.102961+0000","last_clean":"2024-03-16T11:05:03.102961+0000","last_became_active":"2024-03-16T11:03:59.286731+0000","last_became_peered":"2024-03-16T11:03:59.286731+0000","last_unstale":"2024-03-16T11:05:03.102961+0000","last_undegraded":"2024-03-16T11:05:03.102961+0000","last_fullsized":"2024-03-16T11:05:03.102961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:51.443833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999855+0000","last_change":"2024-03-16T11:04:00.294450+0000","last_active":"2024-03-16T11:05:31.999855+0000","last_peered":"2024-03-16T11:05:31.999855+0000","last_clean":"2024-03-16T11:05:31.999855+0000","last_became_active":"2024-03-16T11:04:00.293769+0000","last_became_peered":"2024-03-16T11:04:00.293769+0000","last_unstale":"2024-03-16T11:05:31.999855+0000","last_undegraded":"2024-03-16T11:05:31.999855+0000","last_fullsized":"2024-03-16T11:05:31.999855+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:02.755907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":141,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.597588+0000","last_change":"2024-03-16T11:03:59.286197+0000","last_active":"2024-03-16T11:05:53.597588+0000","last_peered":"2024-03-16T11:05:53.597588+0000","last_clean":"2024-03-16T11:05:53.597588+0000","last_became_active":"2024-03-16T11:03:59.285757+0000","last_became_peered":"2024-03-16T11:03:59.285757+0000","last_unstale":"2024-03-16T11:05:53.597588+0000","last_undegraded":"2024-03-16T11:05:53.597588+0000","last_fullsized":"2024-03-16T11:05:53.597588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:47.868106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002798+0000","last_change":"2024-03-16T11:04:00.296709+0000","last_active":"2024-03-16T11:05:32.002798+0000","last_peered":"2024-03-16T11:05:32.002798+0000","last_clean":"2024-03-16T11:05:32.002798+0000","last_became_active":"2024-03-16T11:04:00.296372+0000","last_became_peered":"2024-03-16T11:04:00.296372+0000","last_unstale":"2024-03-16T11:05:32.002798+0000","last_undegraded":"2024-03-16T11:05:32.002798+0000","last_fullsized":"2024-03-16T11:05:32.002798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:37.948866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108997+0000","last_change":"2024-03-16T11:03:59.283831+0000","last_active":"2024-03-16T11:05:03.108997+0000","last_peered":"2024-03-16T11:05:03.108997+0000","last_clean":"2024-03-16T11:05:03.108997+0000","last_became_active":"2024-03-16T11:03:59.283404+0000","last_became_peered":"2024-03-16T11:03:59.283404+0000","last_unstale":"2024-03-16T11:05:03.108997+0000","last_undegraded":"2024-03-16T11:05:03.108997+0000","last_fullsized":"2024-03-16T11:05:03.108997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:26:34.012597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.984955+0000","last_change":"2024-03-16T11:04:00.299703+0000","last_active":"2024-03-16T11:05:31.984955+0000","last_peered":"2024-03-16T11:05:31.984955+0000","last_clean":"2024-03-16T11:05:31.984955+0000","last_became_active":"2024-03-16T11:04:00.299321+0000","last_became_peered":"2024-03-16T11:04:00.299321+0000","last_unstale":"2024-03-16T11:05:31.984955+0000","last_undegraded":"2024-03-16T11:05:31.984955+0000","last_fullsized":"2024-03-16T11:05:31.984955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.529281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.109510+0000","last_change":"2024-03-16T11:03:59.277837+0000","last_active":"2024-03-16T11:05:03.109510+0000","last_peered":"2024-03-16T11:05:03.109510+0000","last_clean":"2024-03-16T11:05:03.109510+0000","last_became_active":"2024-03-16T11:03:59.277515+0000","last_became_peered":"2024-03-16T11:03:59.277515+0000","last_unstale":"2024-03-16T11:05:03.109510+0000","last_undegraded":"2024-03-16T11:05:03.109510+0000","last_fullsized":"2024-03-16T11:05:03.109510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:05.116990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994100+0000","last_change":"2024-03-16T11:04:00.292998+0000","last_active":"2024-03-16T11:05:31.994100+0000","last_peered":"2024-03-16T11:05:31.994100+0000","last_clean":"2024-03-16T11:05:31.994100+0000","last_became_active":"2024-03-16T11:04:00.292608+0000","last_became_peered":"2024-03-16T11:04:00.292608+0000","last_unstale":"2024-03-16T11:05:31.994100+0000","last_undegraded":"2024-03-16T11:05:31.994100+0000","last_fullsized":"2024-03-16T11:05:31.994100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:41.996826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108505+0000","last_change":"2024-03-16T11:05:03.108505+0000","last_active":"2024-03-16T11:05:03.108505+0000","last_peered":"2024-03-16T11:05:03.108505+0000","last_clean":"2024-03-16T11:05:03.108505+0000","last_became_active":"2024-03-16T11:05:03.108132+0000","last_became_peered":"2024-03-16T11:05:03.108132+0000","last_unstale":"2024-03-16T11:05:03.108505+0000","last_undegraded":"2024-03-16T11:05:03.108505+0000","last_fullsized":"2024-03-16T11:05:03.108505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:50.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.002870+0000","last_change":"2024-03-16T11:04:00.288897+0000","last_active":"2024-03-16T11:05:32.002870+0000","last_peered":"2024-03-16T11:05:32.002870+0000","last_clean":"2024-03-16T11:05:32.002870+0000","last_became_active":"2024-03-16T11:04:00.288493+0000","last_became_peered":"2024-03-16T11:04:00.288493+0000","last_unstale":"2024-03-16T11:05:32.002870+0000","last_undegraded":"2024-03-16T11:05:32.002870+0000","last_fullsized":"2024-03-16T11:05:32.002870+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:31.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105515+0000","last_change":"2024-03-16T11:03:59.291595+0000","last_active":"2024-03-16T11:05:03.105515+0000","last_peered":"2024-03-16T11:05:03.105515+0000","last_clean":"2024-03-16T11:05:03.105515+0000","last_became_active":"2024-03-16T11:03:59.291159+0000","last_became_peered":"2024-03-16T11:03:59.291159+0000","last_unstale":"2024-03-16T11:05:03.105515+0000","last_undegraded":"2024-03-16T11:05:03.105515+0000","last_fullsized":"2024-03-16T11:05:03.105515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:03.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000305+0000","last_change":"2024-03-16T11:05:03.108818+0000","last_active":"2024-03-16T11:05:32.000305+0000","last_peered":"2024-03-16T11:05:32.000305+0000","last_clean":"2024-03-16T11:05:32.000305+0000","last_became_active":"2024-03-16T11:05:03.108479+0000","last_became_peered":"2024-03-16T11:05:03.108479+0000","last_unstale":"2024-03-16T11:05:32.000305+0000","last_undegraded":"2024-03-16T11:05:32.000305+0000","last_fullsized":"2024-03-16T11:05:32.000305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:10:11.329743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.977502+0000","last_change":"2024-03-16T11:04:00.298400+0000","last_active":"2024-03-16T11:05:31.977502+0000","last_peered":"2024-03-16T11:05:31.977502+0000","last_clean":"2024-03-16T11:05:31.977502+0000","last_became_active":"2024-03-16T11:04:00.298026+0000","last_became_peered":"2024-03-16T11:04:00.298026+0000","last_unstale":"2024-03-16T11:05:31.977502+0000","last_undegraded":"2024-03-16T11:05:31.977502+0000","last_fullsized":"2024-03-16T11:05:31.977502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:08.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":136,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:53.225688+0000","last_change":"2024-03-16T11:03:59.287528+0000","last_active":"2024-03-16T11:05:53.225688+0000","last_peered":"2024-03-16T11:05:53.225688+0000","last_clean":"2024-03-16T11:05:53.225688+0000","last_became_active":"2024-03-16T11:03:59.287218+0000","last_became_peered":"2024-03-16T11:03:59.287218+0000","last_unstale":"2024-03-16T11:05:53.225688+0000","last_undegraded":"2024-03-16T11:05:53.225688+0000","last_fullsized":"2024-03-16T11:05:53.225688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:19.803233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.006796+0000","last_change":"2024-03-16T11:05:03.106099+0000","last_active":"2024-03-16T11:05:32.006796+0000","last_peered":"2024-03-16T11:05:32.006796+0000","last_clean":"2024-03-16T11:05:32.006796+0000","last_became_active":"2024-03-16T11:05:03.105412+0000","last_became_peered":"2024-03-16T11:05:03.105412+0000","last_unstale":"2024-03-16T11:05:32.006796+0000","last_undegraded":"2024-03-16T11:05:32.006796+0000","last_fullsized":"2024-03-16T11:05:32.006796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:29:33.213676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.541779+0000","last_change":"2024-03-16T11:03:59.281251+0000","last_active":"2024-03-16T11:05:03.541779+0000","last_peered":"2024-03-16T11:05:03.541779+0000","last_clean":"2024-03-16T11:05:03.541779+0000","last_became_active":"2024-03-16T11:03:59.280830+0000","last_became_peered":"2024-03-16T11:03:59.280830+0000","last_unstale":"2024-03-16T11:05:03.541779+0000","last_undegraded":"2024-03-16T11:05:03.541779+0000","last_fullsized":"2024-03-16T11:05:03.541779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:03:04.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105473+0000","last_change":"2024-03-16T11:02:48.497749+0000","last_active":"2024-03-16T11:05:03.105473+0000","last_peered":"2024-03-16T11:05:03.105473+0000","last_clean":"2024-03-16T11:05:03.105473+0000","last_became_active":"2024-03-16T11:02:48.497157+0000","last_became_peered":"2024-03-16T11:02:48.497157+0000","last_unstale":"2024-03-16T11:05:03.105473+0000","last_undegraded":"2024-03-16T11:05:03.105473+0000","last_fullsized":"2024-03-16T11:05:03.105473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:59:26.309639+0000","last_clean_scrub_stamp":"2024-03-16T10:59:26.309639+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:04.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999743+0000","last_change":"2024-03-16T11:04:00.300821+0000","last_active":"2024-03-16T11:05:31.999743+0000","last_peered":"2024-03-16T11:05:31.999743+0000","last_clean":"2024-03-16T11:05:31.999743+0000","last_became_active":"2024-03-16T11:04:00.300510+0000","last_became_peered":"2024-03-16T11:04:00.300510+0000","last_unstale":"2024-03-16T11:05:31.999743+0000","last_undegraded":"2024-03-16T11:05:31.999743+0000","last_fullsized":"2024-03-16T11:05:31.999743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:57.035167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.105571+0000","last_change":"2024-03-16T11:03:59.283609+0000","last_active":"2024-03-16T11:05:03.105571+0000","last_peered":"2024-03-16T11:05:03.105571+0000","last_clean":"2024-03-16T11:05:03.105571+0000","last_became_active":"2024-03-16T11:03:59.283227+0000","last_became_peered":"2024-03-16T11:03:59.283227+0000","last_unstale":"2024-03-16T11:05:03.105571+0000","last_undegraded":"2024-03-16T11:05:03.105571+0000","last_fullsized":"2024-03-16T11:05:03.105571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:18.904202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.112457+0000","last_change":"2024-03-16T11:05:03.112071+0000","last_active":"2024-03-16T11:05:03.112457+0000","last_peered":"2024-03-16T11:05:03.112457+0000","last_clean":"2024-03-16T11:05:03.112457+0000","last_became_active":"2024-03-16T11:05:03.108434+0000","last_became_peered":"2024-03-16T11:05:03.108434+0000","last_unstale":"2024-03-16T11:05:03.112457+0000","last_undegraded":"2024-03-16T11:05:03.112457+0000","last_fullsized":"2024-03-16T11:05:03.112457+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:24.758886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000550+0000","last_change":"2024-03-16T11:04:00.285134+0000","last_active":"2024-03-16T11:05:32.000550+0000","last_peered":"2024-03-16T11:05:32.000550+0000","last_clean":"2024-03-16T11:05:32.000550+0000","last_became_active":"2024-03-16T11:04:00.284705+0000","last_became_peered":"2024-03-16T11:04:00.284705+0000","last_unstale":"2024-03-16T11:05:32.000550+0000","last_undegraded":"2024-03-16T11:05:32.000550+0000","last_fullsized":"2024-03-16T11:05:32.000550+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:37:38.637970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:54.301595+0000","last_change":"2024-03-16T11:03:59.289999+0000","last_active":"2024-03-16T11:05:54.301595+0000","last_peered":"2024-03-16T11:05:54.301595+0000","last_clean":"2024-03-16T11:05:54.301595+0000","last_became_active":"2024-03-16T11:03:59.289694+0000","last_became_peered":"2024-03-16T11:03:59.289694+0000","last_unstale":"2024-03-16T11:05:54.301595+0000","last_undegraded":"2024-03-16T11:05:54.301595+0000","last_fullsized":"2024-03-16T11:05:54.301595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:01:16.037387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.000119+0000","last_change":"2024-03-16T11:04:00.298148+0000","last_active":"2024-03-16T11:05:32.000119+0000","last_peered":"2024-03-16T11:05:32.000119+0000","last_clean":"2024-03-16T11:05:32.000119+0000","last_became_active":"2024-03-16T11:04:00.297822+0000","last_became_peered":"2024-03-16T11:04:00.297822+0000","last_unstale":"2024-03-16T11:05:32.000119+0000","last_undegraded":"2024-03-16T11:05:32.000119+0000","last_fullsized":"2024-03-16T11:05:32.000119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:09:28.645317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.542005+0000","last_change":"2024-03-16T11:03:59.292207+0000","last_active":"2024-03-16T11:05:03.542005+0000","last_peered":"2024-03-16T11:05:03.542005+0000","last_clean":"2024-03-16T11:05:03.542005+0000","last_became_active":"2024-03-16T11:03:59.291859+0000","last_became_peered":"2024-03-16T11:03:59.291859+0000","last_unstale":"2024-03-16T11:05:03.542005+0000","last_undegraded":"2024-03-16T11:05:03.542005+0000","last_fullsized":"2024-03-16T11:05:03.542005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:55.365720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998794+0000","last_change":"2024-03-16T11:04:00.291261+0000","last_active":"2024-03-16T11:05:31.998794+0000","last_peered":"2024-03-16T11:05:31.998794+0000","last_clean":"2024-03-16T11:05:31.998794+0000","last_became_active":"2024-03-16T11:04:00.290832+0000","last_became_peered":"2024-03-16T11:04:00.290832+0000","last_unstale":"2024-03-16T11:05:31.998794+0000","last_undegraded":"2024-03-16T11:05:31.998794+0000","last_fullsized":"2024-03-16T11:05:31.998794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:10.418808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.100986+0000","last_change":"2024-03-16T11:03:59.288014+0000","last_active":"2024-03-16T11:05:03.100986+0000","last_peered":"2024-03-16T11:05:03.100986+0000","last_clean":"2024-03-16T11:05:03.100986+0000","last_became_active":"2024-03-16T11:03:59.287679+0000","last_became_peered":"2024-03-16T11:03:59.287679+0000","last_unstale":"2024-03-16T11:05:03.100986+0000","last_undegraded":"2024-03-16T11:05:03.100986+0000","last_fullsized":"2024-03-16T11:05:03.100986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:16:39.156224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.960063+0000","last_change":"2024-03-16T11:04:00.288047+0000","last_active":"2024-03-16T11:05:31.960063+0000","last_peered":"2024-03-16T11:05:31.960063+0000","last_clean":"2024-03-16T11:05:31.960063+0000","last_became_active":"2024-03-16T11:04:00.287619+0000","last_became_peered":"2024-03-16T11:04:00.287619+0000","last_unstale":"2024-03-16T11:05:31.960063+0000","last_undegraded":"2024-03-16T11:05:31.960063+0000","last_fullsized":"2024-03-16T11:05:31.960063+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:43.608349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.106759+0000","last_change":"2024-03-16T11:03:59.288439+0000","last_active":"2024-03-16T11:05:03.106759+0000","last_peered":"2024-03-16T11:05:03.106759+0000","last_clean":"2024-03-16T11:05:03.106759+0000","last_became_active":"2024-03-16T11:03:59.288122+0000","last_became_peered":"2024-03-16T11:03:59.288122+0000","last_unstale":"2024-03-16T11:05:03.106759+0000","last_undegraded":"2024-03-16T11:05:03.106759+0000","last_fullsized":"2024-03-16T11:05:03.106759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:50:25.499608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998940+0000","last_change":"2024-03-16T11:04:00.290522+0000","last_active":"2024-03-16T11:05:31.998940+0000","last_peered":"2024-03-16T11:05:31.998940+0000","last_clean":"2024-03-16T11:05:31.998940+0000","last_became_active":"2024-03-16T11:04:00.290175+0000","last_became_peered":"2024-03-16T11:04:00.290175+0000","last_unstale":"2024-03-16T11:05:31.998940+0000","last_undegraded":"2024-03-16T11:05:31.998940+0000","last_fullsized":"2024-03-16T11:05:31.998940+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:34.113278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548523+0000","last_change":"2024-03-16T11:03:59.289682+0000","last_active":"2024-03-16T11:05:03.548523+0000","last_peered":"2024-03-16T11:05:03.548523+0000","last_clean":"2024-03-16T11:05:03.548523+0000","last_became_active":"2024-03-16T11:03:59.289367+0000","last_became_peered":"2024-03-16T11:03:59.289367+0000","last_unstale":"2024-03-16T11:05:03.548523+0000","last_undegraded":"2024-03-16T11:05:03.548523+0000","last_fullsized":"2024-03-16T11:05:03.548523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:42:48.627116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994427+0000","last_change":"2024-03-16T11:04:00.287685+0000","last_active":"2024-03-16T11:05:31.994427+0000","last_peered":"2024-03-16T11:05:31.994427+0000","last_clean":"2024-03-16T11:05:31.994427+0000","last_became_active":"2024-03-16T11:04:00.287367+0000","last_became_peered":"2024-03-16T11:04:00.287367+0000","last_unstale":"2024-03-16T11:05:31.994427+0000","last_undegraded":"2024-03-16T11:05:31.994427+0000","last_fullsized":"2024-03-16T11:05:31.994427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:25:52.741651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.108923+0000","last_change":"2024-03-16T11:03:59.286293+0000","last_active":"2024-03-16T11:05:03.108923+0000","last_peered":"2024-03-16T11:05:03.108923+0000","last_clean":"2024-03-16T11:05:03.108923+0000","last_became_active":"2024-03-16T11:03:59.285817+0000","last_became_peered":"2024-03-16T11:03:59.285817+0000","last_unstale":"2024-03-16T11:05:03.108923+0000","last_undegraded":"2024-03-16T11:05:03.108923+0000","last_fullsized":"2024-03-16T11:05:03.108923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:38:12.093762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008486+0000","last_change":"2024-03-16T11:04:00.299391+0000","last_active":"2024-03-16T11:05:32.008486+0000","last_peered":"2024-03-16T11:05:32.008486+0000","last_clean":"2024-03-16T11:05:32.008486+0000","last_became_active":"2024-03-16T11:04:00.298931+0000","last_became_peered":"2024-03-16T11:04:00.298931+0000","last_unstale":"2024-03-16T11:05:32.008486+0000","last_undegraded":"2024-03-16T11:05:32.008486+0000","last_fullsized":"2024-03-16T11:05:32.008486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:12:11.900154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.548534+0000","last_change":"2024-03-16T11:03:59.292319+0000","last_active":"2024-03-16T11:05:03.548534+0000","last_peered":"2024-03-16T11:05:03.548534+0000","last_clean":"2024-03-16T11:05:03.548534+0000","last_became_active":"2024-03-16T11:03:59.291953+0000","last_became_peered":"2024-03-16T11:03:59.291953+0000","last_unstale":"2024-03-16T11:05:03.548534+0000","last_undegraded":"2024-03-16T11:05:03.548534+0000","last_fullsized":"2024-03-16T11:05:03.548534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:41:04.615726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008131+0000","last_change":"2024-03-16T11:04:00.296544+0000","last_active":"2024-03-16T11:05:32.008131+0000","last_peered":"2024-03-16T11:05:32.008131+0000","last_clean":"2024-03-16T11:05:32.008131+0000","last_became_active":"2024-03-16T11:04:00.296078+0000","last_became_peered":"2024-03-16T11:04:00.296078+0000","last_unstale":"2024-03-16T11:05:32.008131+0000","last_undegraded":"2024-03-16T11:05:32.008131+0000","last_fullsized":"2024-03-16T11:05:32.008131+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:56:22.846734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103250+0000","last_change":"2024-03-16T11:03:59.287172+0000","last_active":"2024-03-16T11:05:03.103250+0000","last_peered":"2024-03-16T11:05:03.103250+0000","last_clean":"2024-03-16T11:05:03.103250+0000","last_became_active":"2024-03-16T11:03:59.286785+0000","last_became_peered":"2024-03-16T11:03:59.286785+0000","last_unstale":"2024-03-16T11:05:03.103250+0000","last_undegraded":"2024-03-16T11:05:03.103250+0000","last_fullsized":"2024-03-16T11:05:03.103250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:17:55.584618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990540+0000","last_change":"2024-03-16T11:05:03.111534+0000","last_active":"2024-03-16T11:05:31.990540+0000","last_peered":"2024-03-16T11:05:31.990540+0000","last_clean":"2024-03-16T11:05:31.990540+0000","last_became_active":"2024-03-16T11:05:03.107517+0000","last_became_peered":"2024-03-16T11:05:03.107517+0000","last_unstale":"2024-03-16T11:05:31.990540+0000","last_undegraded":"2024-03-16T11:05:31.990540+0000","last_fullsized":"2024-03-16T11:05:31.990540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:07:51.828833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:03.103117+0000","last_change":"2024-03-16T11:03:59.288822+0000","last_active":"2024-03-16T11:05:03.103117+0000","last_peered":"2024-03-16T11:05:03.103117+0000","last_clean":"2024-03-16T11:05:03.103117+0000","last_became_active":"2024-03-16T11:03:59.288516+0000","last_became_peered":"2024-03-16T11:03:59.288516+0000","last_unstale":"2024-03-16T11:05:03.103117+0000","last_undegraded":"2024-03-16T11:05:03.103117+0000","last_fullsized":"2024-03-16T11:05:03.103117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:58.260933+0000","last_clean_scrub_stamp":"2024-03-16T11:03:58.260933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:32:38.862476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.003380+0000","last_change":"2024-03-16T11:04:00.296117+0000","last_active":"2024-03-16T11:05:32.003380+0000","last_peered":"2024-03-16T11:05:32.003380+0000","last_clean":"2024-03-16T11:05:32.003380+0000","last_became_active":"2024-03-16T11:04:00.295664+0000","last_became_peered":"2024-03-16T11:04:00.295664+0000","last_unstale":"2024-03-16T11:05:32.003380+0000","last_undegraded":"2024-03-16T11:05:32.003380+0000","last_fullsized":"2024-03-16T11:05:32.003380+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:05.448616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999611+0000","last_change":"2024-03-16T11:04:00.295035+0000","last_active":"2024-03-16T11:05:31.999611+0000","last_peered":"2024-03-16T11:05:31.999611+0000","last_clean":"2024-03-16T11:05:31.999611+0000","last_became_active":"2024-03-16T11:04:00.294689+0000","last_became_peered":"2024-03-16T11:04:00.294689+0000","last_unstale":"2024-03-16T11:05:31.999611+0000","last_undegraded":"2024-03-16T11:05:31.999611+0000","last_fullsized":"2024-03-16T11:05:31.999611+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:55.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998519+0000","last_change":"2024-03-16T11:04:00.292831+0000","last_active":"2024-03-16T11:05:31.998519+0000","last_peered":"2024-03-16T11:05:31.998519+0000","last_clean":"2024-03-16T11:05:31.998519+0000","last_became_active":"2024-03-16T11:04:00.292387+0000","last_became_peered":"2024-03-16T11:04:00.292387+0000","last_unstale":"2024-03-16T11:05:31.998519+0000","last_undegraded":"2024-03-16T11:05:31.998519+0000","last_fullsized":"2024-03-16T11:05:31.998519+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:14:53.273535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.970952+0000","last_change":"2024-03-16T11:04:00.291267+0000","last_active":"2024-03-16T11:05:31.970952+0000","last_peered":"2024-03-16T11:05:31.970952+0000","last_clean":"2024-03-16T11:05:31.970952+0000","last_became_active":"2024-03-16T11:04:00.290911+0000","last_became_peered":"2024-03-16T11:04:00.290911+0000","last_unstale":"2024-03-16T11:05:31.970952+0000","last_undegraded":"2024-03-16T11:05:31.970952+0000","last_fullsized":"2024-03-16T11:05:31.970952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:35.285167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999212+0000","last_change":"2024-03-16T11:04:00.300470+0000","last_active":"2024-03-16T11:05:31.999212+0000","last_peered":"2024-03-16T11:05:31.999212+0000","last_clean":"2024-03-16T11:05:31.999212+0000","last_became_active":"2024-03-16T11:04:00.300107+0000","last_became_peered":"2024-03-16T11:04:00.300107+0000","last_unstale":"2024-03-16T11:05:31.999212+0000","last_undegraded":"2024-03-16T11:05:31.999212+0000","last_fullsized":"2024-03-16T11:05:31.999212+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:18.718749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.001171+0000","last_change":"2024-03-16T11:04:00.300857+0000","last_active":"2024-03-16T11:05:32.001171+0000","last_peered":"2024-03-16T11:05:32.001171+0000","last_clean":"2024-03-16T11:05:32.001171+0000","last_became_active":"2024-03-16T11:04:00.300551+0000","last_became_peered":"2024-03-16T11:04:00.300551+0000","last_unstale":"2024-03-16T11:05:32.001171+0000","last_undegraded":"2024-03-16T11:05:32.001171+0000","last_fullsized":"2024-03-16T11:05:32.001171+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:24.362561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.996929+0000","last_change":"2024-03-16T11:05:03.110781+0000","last_active":"2024-03-16T11:05:31.996929+0000","last_peered":"2024-03-16T11:05:31.996929+0000","last_clean":"2024-03-16T11:05:31.996929+0000","last_became_active":"2024-03-16T11:05:03.110375+0000","last_became_peered":"2024-03-16T11:05:03.110375+0000","last_unstale":"2024-03-16T11:05:31.996929+0000","last_undegraded":"2024-03-16T11:05:31.996929+0000","last_fullsized":"2024-03-16T11:05:31.996929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:28.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.998128+0000","last_change":"2024-03-16T11:04:00.294943+0000","last_active":"2024-03-16T11:05:31.998128+0000","last_peered":"2024-03-16T11:05:31.998128+0000","last_clean":"2024-03-16T11:05:31.998128+0000","last_became_active":"2024-03-16T11:04:00.294637+0000","last_became_peered":"2024-03-16T11:04:00.294637+0000","last_unstale":"2024-03-16T11:05:31.998128+0000","last_undegraded":"2024-03-16T11:05:31.998128+0000","last_fullsized":"2024-03-16T11:05:31.998128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:03:09.716753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.990054+0000","last_change":"2024-03-16T11:04:00.300737+0000","last_active":"2024-03-16T11:05:31.990054+0000","last_peered":"2024-03-16T11:05:31.990054+0000","last_clean":"2024-03-16T11:05:31.990054+0000","last_became_active":"2024-03-16T11:04:00.300410+0000","last_became_peered":"2024-03-16T11:04:00.300410+0000","last_unstale":"2024-03-16T11:05:31.990054+0000","last_undegraded":"2024-03-16T11:05:31.990054+0000","last_fullsized":"2024-03-16T11:05:31.990054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:18.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.971404+0000","last_change":"2024-03-16T11:04:00.292958+0000","last_active":"2024-03-16T11:05:31.971404+0000","last_peered":"2024-03-16T11:05:31.971404+0000","last_clean":"2024-03-16T11:05:31.971404+0000","last_became_active":"2024-03-16T11:04:00.292599+0000","last_became_peered":"2024-03-16T11:04:00.292599+0000","last_unstale":"2024-03-16T11:05:31.971404+0000","last_undegraded":"2024-03-16T11:05:31.971404+0000","last_fullsized":"2024-03-16T11:05:31.971404+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:20:46.370597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.994683+0000","last_change":"2024-03-16T11:04:00.291381+0000","last_active":"2024-03-16T11:05:31.994683+0000","last_peered":"2024-03-16T11:05:31.994683+0000","last_clean":"2024-03-16T11:05:31.994683+0000","last_became_active":"2024-03-16T11:04:00.291046+0000","last_became_peered":"2024-03-16T11:04:00.291046+0000","last_unstale":"2024-03-16T11:05:31.994683+0000","last_undegraded":"2024-03-16T11:05:31.994683+0000","last_fullsized":"2024-03-16T11:05:31.994683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:08.185733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:32.008021+0000","last_change":"2024-03-16T11:04:00.301381+0000","last_active":"2024-03-16T11:05:32.008021+0000","last_peered":"2024-03-16T11:05:32.008021+0000","last_clean":"2024-03-16T11:05:32.008021+0000","last_became_active":"2024-03-16T11:04:00.300990+0000","last_became_peered":"2024-03-16T11:04:00.300990+0000","last_unstale":"2024-03-16T11:05:32.008021+0000","last_undegraded":"2024-03-16T11:05:32.008021+0000","last_fullsized":"2024-03-16T11:05:32.008021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:07:43.775066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999944+0000","last_change":"2024-03-16T11:04:00.298220+0000","last_active":"2024-03-16T11:05:31.999944+0000","last_peered":"2024-03-16T11:05:31.999944+0000","last_clean":"2024-03-16T11:05:31.999944+0000","last_became_active":"2024-03-16T11:04:00.297776+0000","last_became_peered":"2024-03-16T11:04:00.297776+0000","last_unstale":"2024-03-16T11:05:31.999944+0000","last_undegraded":"2024-03-16T11:05:31.999944+0000","last_fullsized":"2024-03-16T11:05:31.999944+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:59:40.387309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T11:05:31.999183+0000","last_change":"2024-03-16T11:04:00.292988+0000","last_active":"2024-03-16T11:05:31.999183+0000","last_peered":"2024-03-16T11:05:31.999183+0000","last_clean":"2024-03-16T11:05:31.999183+0000","last_became_active":"2024-03-16T11:04:00.292655+0000","last_became_peered":"2024-03-16T11:04:00.292655+0000","last_unstale":"2024-03-16T11:05:31.999183+0000","last_undegraded":"2024-03-16T11:05:31.999183+0000","last_fullsized":"2024-03-16T11:05:31.999183+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:03:59.264745+0000","last_clean_scrub_stamp":"2024-03-16T11:03:59.264745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:33:43.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71813,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":702,"num_read_kb":424,"num_write":176,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":214453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285604,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":471093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547242,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1683456,"data_stored":1128327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743472,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":471307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005108,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":471205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1593344,"data_stored":1063298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430270,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":470813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":536783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":471967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":539017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":471007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":472158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1601536,"data_stored":1063217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:05:58.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 -- 172.21.15.78:0/2378921376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa8104c10 msgr2=0x7fafa8107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/2378921376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa8104c10 0x7fafa8107000 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009900 tx=0x7faf9c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 -- 172.21.15.78:0/2378921376 shutdown_connections 2024-03-16T11:05:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/2378921376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8107540 0x7fafa8109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/2378921376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa8104c10 0x7fafa8107000 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/2378921376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa81022e0 0x7fafa81046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 -- 172.21.15.78:0/2378921376 >> 172.21.15.78:0/2378921376 conn(0x7fafa80fbc70 msgr2=0x7fafa80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 -- 172.21.15.78:0/2378921376 shutdown_connections 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.359+0000 7fafaf0ff640 1 -- 172.21.15.78:0/2378921376 wait complete. 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 Processor -- start 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 -- start start 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa81022e0 0x7fafa8199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa8104c10 0x7fafa819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8107540 0x7fafa81a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fafa810d320 con 0x7fafa81022e0 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafa810d1a0 con 0x7fafa8107540 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafaf0ff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafa810d4a0 con 0x7fafa8104c10 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa81022e0 0x7fafa8199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa81022e0 0x7fafa8199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54828/0 (socket says 172.21.15.78:54828) 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 -- 172.21.15.78:0/1030217967 learned_addr learned my addr 172.21.15.78:0/1030217967 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafad69c640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8107540 0x7fafa81a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7fafa7fff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa8104c10 0x7fafa819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 -- 172.21.15.78:0/1030217967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa8104c10 msgr2=0x7fafa819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa8104c10 0x7fafa819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 -- 172.21.15.78:0/1030217967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8107540 msgr2=0x7fafa81a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8107540 0x7fafa81a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.360+0000 7faface9b640 1 -- 172.21.15.78:0/1030217967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa81a18e0 con 0x7fafa81022e0 2024-03-16T11:05:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.361+0000 7faface9b640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa81022e0 0x7fafa8199ca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7faf9800e9f0 tx=0x7faf9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.361+0000 7fafa5ffb640 1 -- 172.21.15.78:0/1030217967 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9800cd80 con 0x7fafa81022e0 2024-03-16T11:05:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.361+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa81a1bd0 con 0x7fafa81022e0 2024-03-16T11:05:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.361+0000 7fafa5ffb640 1 -- 172.21.15.78:0/1030217967 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf980054d0 con 0x7fafa81022e0 2024-03-16T11:05:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.361+0000 7fafa5ffb640 1 -- 172.21.15.78:0/1030217967 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98010490 con 0x7fafa81022e0 2024-03-16T11:05:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.361+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa81a20f0 con 0x7fafa81022e0 2024-03-16T11:05:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.362+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa8109030 con 0x7fafa81022e0 2024-03-16T11:05:58.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.363+0000 7fafa5ffb640 1 -- 172.21.15.78:0/1030217967 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7faf98005020 con 0x7fafa81022e0 2024-03-16T11:05:58.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.363+0000 7fafa5ffb640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf900a2890 0x7faf900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.363+0000 7fafa5ffb640 1 -- 172.21.15.78:0/1030217967 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faf980cef50 con 0x7fafa81022e0 2024-03-16T11:05:58.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.363+0000 7fafa7fff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf900a2890 0x7faf900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.364+0000 7fafa7fff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf900a2890 0x7faf900a4d50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faf9c005ab0 tx=0x7faf9c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.368+0000 7fafa5ffb640 1 -- 172.21.15.78:0/1030217967 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf9809be90 con 0x7fafa81022e0 2024-03-16T11:05:58.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:58 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3011666219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:58.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:58 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3011666219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:05:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.547+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafa802d360 con 0x7fafa81022e0 2024-03-16T11:05:58.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.549+0000 7fafa5ffb640 1 -- 172.21.15.78:0/1030217967 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7faf9809b790 con 0x7fafa81022e0 2024-03-16T11:05:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:58.550 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf900a2890 msgr2=0x7faf900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf900a2890 0x7faf900a4d50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faf9c005ab0 tx=0x7faf9c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa81022e0 msgr2=0x7fafa8199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa81022e0 0x7fafa8199ca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7faf9800e9f0 tx=0x7faf9800edf0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 shutdown_connections 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa8107540 0x7fafa81a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7faf900a2890 0x7faf900a4d50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa8104c10 0x7fafa819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 --2- 172.21.15.78:0/1030217967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafa81022e0 0x7fafa8199ca0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 >> 172.21.15.78:0/1030217967 conn(0x7fafa80fbc70 msgr2=0x7fafa8101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 shutdown_connections 2024-03-16T11:05:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.551+0000 7fafaf0ff640 1 -- 172.21.15.78:0/1030217967 wait complete. 2024-03-16T11:05:58.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:05:58.563 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:05:58.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 -- 172.21.15.78:0/3643228514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1018d0 msgr2=0x7ff43c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 --2- 172.21.15.78:0/3643228514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1018d0 0x7ff43c101d50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff428009900 tx=0x7ff42802bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 -- 172.21.15.78:0/3643228514 shutdown_connections 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 --2- 172.21.15.78:0/3643228514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff43c102290 0x7ff43c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 --2- 172.21.15.78:0/3643228514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1018d0 0x7ff43c101d50 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 --2- 172.21.15.78:0/3643228514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff43c1006e0 0x7ff43c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 -- 172.21.15.78:0/3643228514 >> 172.21.15.78:0/3643228514 conn(0x7ff43c0fbc70 msgr2=0x7ff43c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 -- 172.21.15.78:0/3643228514 shutdown_connections 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 -- 172.21.15.78:0/3643228514 wait complete. 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.667+0000 7ff44300c640 1 Processor -- start 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 -- start start 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1006e0 0x7ff43c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff43c1018d0 0x7ff43c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff43c102290 0x7ff43c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c10c210 con 0x7ff43c1006e0 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c10c090 con 0x7ff43c102290 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c10c390 con 0x7ff43c1018d0 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1006e0 0x7ff43c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1006e0 0x7ff43c197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54832/0 (socket says 172.21.15.78:54832) 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 -- 172.21.15.78:0/302251193 learned_addr learned my addr 172.21.15.78:0/302251193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff43bfff640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff43c1018d0 0x7ff43c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff4415a9640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff43c102290 0x7ff43c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 -- 172.21.15.78:0/302251193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff43c1018d0 msgr2=0x7ff43c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff43c1018d0 0x7ff43c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 -- 172.21.15.78:0/302251193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff43c102290 msgr2=0x7ff43c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff43c102290 0x7ff43c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 -- 172.21.15.78:0/302251193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c19f680 con 0x7ff43c1006e0 2024-03-16T11:05:58.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff4415a9640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff43c102290 0x7ff43c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:05:58.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff440da8640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1006e0 0x7ff43c197a60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff43000e9c0 tx=0x7ff43000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff439ffb640 1 -- 172.21.15.78:0/302251193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43000cd80 con 0x7ff43c1006e0 2024-03-16T11:05:58.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff439ffb640 1 -- 172.21.15.78:0/302251193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4300035f0 con 0x7ff43c1006e0 2024-03-16T11:05:58.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43c19f970 con 0x7ff43c1006e0 2024-03-16T11:05:58.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.668+0000 7ff439ffb640 1 -- 172.21.15.78:0/302251193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430010690 con 0x7ff43c1006e0 2024-03-16T11:05:58.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.669+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43c19fe40 con 0x7ff43c1006e0 2024-03-16T11:05:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.670+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff43c1a0010 con 0x7ff43c1006e0 2024-03-16T11:05:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.671+0000 7ff439ffb640 1 -- 172.21.15.78:0/302251193 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7ff4300037a0 con 0x7ff43c1006e0 2024-03-16T11:05:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.671+0000 7ff439ffb640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff42c0a2890 0x7ff42c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.671+0000 7ff439ffb640 1 -- 172.21.15.78:0/302251193 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff4300cf1a0 con 0x7ff43c1006e0 2024-03-16T11:05:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.671+0000 7ff43bfff640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff42c0a2890 0x7ff42c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.672+0000 7ff43bfff640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff42c0a2890 0x7ff42c0a4d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff428009230 tx=0x7ff428005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.675+0000 7ff439ffb640 1 -- 172.21.15.78:0/302251193 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff43009c0e0 con 0x7ff43c1006e0 2024-03-16T11:05:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.848+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff43c1a01e0 con 0x7ff43c1006e0 2024-03-16T11:05:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.849+0000 7ff439ffb640 1 -- 172.21.15.78:0/302251193 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7ff43009b9e0 con 0x7ff43c1006e0 2024-03-16T11:05:58.850 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:05:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff42c0a2890 msgr2=0x7ff42c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff42c0a2890 0x7ff42c0a4d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff428009230 tx=0x7ff428005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1006e0 msgr2=0x7ff43c197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1006e0 0x7ff43c197a60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff43000e9c0 tx=0x7ff43000ee80 comp rx=0 tx=0).stop 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 shutdown_connections 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff43c102290 0x7ff43c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff42c0a2890 0x7ff42c0a4d50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff43c1018d0 0x7ff43c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 --2- 172.21.15.78:0/302251193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff43c1006e0 0x7ff43c197a60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 >> 172.21.15.78:0/302251193 conn(0x7ff43c0fbc70 msgr2=0x7ff43c069e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:58.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 shutdown_connections 2024-03-16T11:05:58.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.851+0000 7ff44300c640 1 -- 172.21.15.78:0/302251193 wait complete. 2024-03-16T11:05:58.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 -- 172.21.15.78:0/569404013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81022a0 msgr2=0x7f44e8108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 --2- 172.21.15.78:0/569404013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81022a0 0x7f44e8108b30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009ca0 tx=0x7f44e00345f0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 -- 172.21.15.78:0/569404013 shutdown_connections 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 --2- 172.21.15.78:0/569404013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81022a0 0x7f44e8108b30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 --2- 172.21.15.78:0/569404013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e81018e0 0x7f44e8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 --2- 172.21.15.78:0/569404013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e81006f0 0x7f44e8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 -- 172.21.15.78:0/569404013 >> 172.21.15.78:0/569404013 conn(0x7f44e80fbc80 msgr2=0x7f44e80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 -- 172.21.15.78:0/569404013 shutdown_connections 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 -- 172.21.15.78:0/569404013 wait complete. 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 Processor -- start 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.967+0000 7f44ee84b640 1 -- start start 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81006f0 0x7f44e81957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e81018e0 0x7f44e8195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e81022a0 0x7f44e819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f44e810b120 con 0x7f44e81006f0 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44e810afa0 con 0x7f44e81022a0 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44e810b2a0 con 0x7f44e81018e0 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81006f0 0x7f44e81957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81006f0 0x7f44e81957d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54850/0 (socket says 172.21.15.78:54850) 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 -- 172.21.15.78:0/2460923216 learned_addr learned my addr 172.21.15.78:0/2460923216 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e77fe640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e81018e0 0x7f44e8195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ecde8640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e81022a0 0x7f44e819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 -- 172.21.15.78:0/2460923216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e81018e0 msgr2=0x7f44e8195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e81018e0 0x7f44e8195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 -- 172.21.15.78:0/2460923216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e81022a0 msgr2=0x7f44e819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e81022a0 0x7f44e819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 -- 172.21.15.78:0/2460923216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e819d3f0 con 0x7f44e81006f0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e7fff640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81006f0 0x7f44e81957d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f44d80099c0 tx=0x7f44d8009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e57fa640 1 -- 172.21.15.78:0/2460923216 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d800cc60 con 0x7f44e81006f0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e57fa640 1 -- 172.21.15.78:0/2460923216 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44d800ce00 con 0x7f44e81006f0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44e57fa640 1 -- 172.21.15.78:0/2460923216 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d8017660 con 0x7f44e81006f0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e819d6e0 con 0x7f44e81006f0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.968+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e819dc30 con 0x7f44e81006f0 2024-03-16T11:05:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.969+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e8107a90 con 0x7f44e81006f0 2024-03-16T11:05:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.970+0000 7f44e57fa640 1 -- 172.21.15.78:0/2460923216 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f44d801e020 con 0x7f44e81006f0 2024-03-16T11:05:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.971+0000 7f44e57fa640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f44d00a2890 0x7f44d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.971+0000 7f44e77fe640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f44d00a2890 0x7f44d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.971+0000 7f44e57fa640 1 -- 172.21.15.78:0/2460923216 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f44d80ce8e0 con 0x7f44e81006f0 2024-03-16T11:05:58.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.971+0000 7f44e77fe640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f44d00a2890 0x7f44d00a4d50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f44e8196ce0 tx=0x7f44dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:58.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:58.975+0000 7f44e57fa640 1 -- 172.21.15.78:0/2460923216 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44d809b820 con 0x7f44e81006f0 2024-03-16T11:05:59.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.154+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f44e80623e0 con 0x7f44e81006f0 2024-03-16T11:05:59.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.154+0000 7f44e57fa640 1 -- 172.21.15.78:0/2460923216 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f44d809b120 con 0x7f44e81006f0 2024-03-16T11:05:59.155 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:05:59.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.156+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f44d00a2890 msgr2=0x7f44d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.156+0000 7f44ee84b640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f44d00a2890 0x7f44d00a4d50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f44e8196ce0 tx=0x7f44dc005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81006f0 msgr2=0x7f44e81957d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81006f0 0x7f44e81957d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f44d80099c0 tx=0x7f44d8009e80 comp rx=0 tx=0).stop 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 shutdown_connections 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e81022a0 0x7f44e819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f44d00a2890 0x7f44d00a4d50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e81018e0 0x7f44e8195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 --2- 172.21.15.78:0/2460923216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44e81006f0 0x7f44e81957d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 >> 172.21.15.78:0/2460923216 conn(0x7f44e80fbc80 msgr2=0x7f44e8104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 shutdown_connections 2024-03-16T11:05:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.157+0000 7f44ee84b640 1 -- 172.21.15.78:0/2460923216 wait complete. 2024-03-16T11:05:59.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:59 smithi078 ceph-mon[32235]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:59 smithi078 ceph-mon[32235]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:59 smithi078 ceph-mon[32235]: pgmap v348: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-16T11:05:59.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:59 smithi078 ceph-mon[32235]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1030217967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:59.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:05:59 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/302251193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:59.168 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:05:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 -- 172.21.15.78:0/3003586078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501006e0 msgr2=0x7f7e50100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 --2- 172.21.15.78:0/3003586078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501006e0 0x7f7e50100b00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009900 tx=0x7f7e4002bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 -- 172.21.15.78:0/3003586078 shutdown_connections 2024-03-16T11:05:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 --2- 172.21.15.78:0/3003586078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e50102290 0x7f7e50108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 --2- 172.21.15.78:0/3003586078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e501018d0 0x7f7e50101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 --2- 172.21.15.78:0/3003586078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501006e0 0x7f7e50100b00 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 -- 172.21.15.78:0/3003586078 >> 172.21.15.78:0/3003586078 conn(0x7f7e500fbc70 msgr2=0x7f7e500fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 -- 172.21.15.78:0/3003586078 shutdown_connections 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.272+0000 7f7e54f40640 1 -- 172.21.15.78:0/3003586078 wait complete. 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 Processor -- start 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 -- start start 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501006e0 0x7f7e50197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501018d0 0x7f7e50197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e50102290 0x7f7e5019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5010c210 con 0x7f7e501018d0 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5010c090 con 0x7f7e50102290 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e54f40640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5010c390 con 0x7f7e501006e0 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e47fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501018d0 0x7f7e50197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e47fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501018d0 0x7f7e50197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54856/0 (socket says 172.21.15.78:54856) 2024-03-16T11:05:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e47fff640 1 -- 172.21.15.78:0/4168030053 learned_addr learned my addr 172.21.15.78:0/4168030053 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e4ed9d640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e50102290 0x7f7e5019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.273+0000 7f7e4e59c640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501006e0 0x7f7e50197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e47fff640 1 -- 172.21.15.78:0/4168030053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501006e0 msgr2=0x7f7e50197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e47fff640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501006e0 0x7f7e50197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e47fff640 1 -- 172.21.15.78:0/4168030053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e50102290 msgr2=0x7f7e5019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e47fff640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e50102290 0x7f7e5019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e47fff640 1 -- 172.21.15.78:0/4168030053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5019f6c0 con 0x7f7e501018d0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e47fff640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501018d0 0x7f7e50197fe0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800e9f0 tx=0x7f7e3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e477fe640 1 -- 172.21.15.78:0/4168030053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3800cd80 con 0x7f7e501018d0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e477fe640 1 -- 172.21.15.78:0/4168030053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e380108e0 con 0x7f7e501018d0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5019f9b0 con 0x7f7e501018d0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e477fe640 1 -- 172.21.15.78:0/4168030053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38018920 con 0x7f7e501018d0 2024-03-16T11:05:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.274+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5019ff00 con 0x7f7e501018d0 2024-03-16T11:05:59.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.275+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e501a0130 con 0x7f7e501018d0 2024-03-16T11:05:59.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.276+0000 7f7e477fe640 1 -- 172.21.15.78:0/4168030053 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f7e38010a90 con 0x7f7e501018d0 2024-03-16T11:05:59.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.277+0000 7f7e477fe640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7e3c0a2890 0x7f7e3c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.277+0000 7f7e4e59c640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7e3c0a2890 0x7f7e3c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.277+0000 7f7e477fe640 1 -- 172.21.15.78:0/4168030053 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7e380ceee0 con 0x7f7e501018d0 2024-03-16T11:05:59.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.277+0000 7f7e4e59c640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7e3c0a2890 0x7f7e3c0a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005ab0 tx=0x7f7e40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:59.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.281+0000 7f7e477fe640 1 -- 172.21.15.78:0/4168030053 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e3809b7d0 con 0x7f7e501018d0 2024-03-16T11:05:59.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.460+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7e501a0300 con 0x7f7e501018d0 2024-03-16T11:05:59.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.460+0000 7f7e477fe640 1 -- 172.21.15.78:0/4168030053 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f7e3809f870 con 0x7f7e501018d0 2024-03-16T11:05:59.461 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7e3c0a2890 msgr2=0x7f7e3c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7e3c0a2890 0x7f7e3c0a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005ab0 tx=0x7f7e40005fd0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501018d0 msgr2=0x7f7e50197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501018d0 0x7f7e50197fe0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800e9f0 tx=0x7f7e3800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 shutdown_connections 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e50102290 0x7f7e5019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e501018d0 0x7f7e50197fe0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7e3c0a2890 0x7f7e3c0a4d50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 --2- 172.21.15.78:0/4168030053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501006e0 0x7f7e50197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 >> 172.21.15.78:0/4168030053 conn(0x7f7e500fbc70 msgr2=0x7f7e500689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:59.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 shutdown_connections 2024-03-16T11:05:59.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.462+0000 7f7e54f40640 1 -- 172.21.15.78:0/4168030053 wait complete. 2024-03-16T11:05:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:59 smithi098 ceph-mon[35848]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:59 smithi098 ceph-mon[35848]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:59 smithi098 ceph-mon[35848]: pgmap v348: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-16T11:05:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:59 smithi098 ceph-mon[35848]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1030217967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:59.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:05:59 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/302251193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:59.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:05:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:59 smithi153 ceph-mon[35521]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:59 smithi153 ceph-mon[35521]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:59 smithi153 ceph-mon[35521]: pgmap v348: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-16T11:05:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:59 smithi153 ceph-mon[35521]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:05:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1030217967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:05:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:05:59 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/302251193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:05:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.574+0000 7f96c8edc640 1 -- 172.21.15.78:0/2634986821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c41009f0 msgr2=0x7f96c4100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.574+0000 7f96c8edc640 1 --2- 172.21.15.78:0/2634986821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c41009f0 0x7f96c4100e70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009900 tx=0x7f96b802bc30 comp rx=0 tx=0).stop 2024-03-16T11:05:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 -- 172.21.15.78:0/2634986821 shutdown_connections 2024-03-16T11:05:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 --2- 172.21.15.78:0/2634986821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c41013b0 0x7f96c4108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 --2- 172.21.15.78:0/2634986821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c41009f0 0x7f96c4100e70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 --2- 172.21.15.78:0/2634986821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96c4101cc0 0x7f96c41020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 -- 172.21.15.78:0/2634986821 >> 172.21.15.78:0/2634986821 conn(0x7f96c40fb420 msgr2=0x7f96c40fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 -- 172.21.15.78:0/2634986821 shutdown_connections 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 -- 172.21.15.78:0/2634986821 wait complete. 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 Processor -- start 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.575+0000 7f96c8edc640 1 -- start start 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96c41009f0 0x7f96c4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c41013b0 0x7f96c4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c4101cc0 0x7f96c419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f96c410c270 con 0x7f96c4101cc0 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96c410c0f0 con 0x7f96c41013b0 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96c410c3f0 con 0x7f96c41009f0 2024-03-16T11:05:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c4101cc0 0x7f96c419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c4101cc0 0x7f96c419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54860/0 (socket says 172.21.15.78:54860) 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 -- 172.21.15.78:0/1657183488 learned_addr learned my addr 172.21.15.78:0/1657183488 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c1d9b640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c41013b0 0x7f96c4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c259c640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96c41009f0 0x7f96c4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 -- 172.21.15.78:0/1657183488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96c41009f0 msgr2=0x7f96c4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96c41009f0 0x7f96c4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 -- 172.21.15.78:0/1657183488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c41013b0 msgr2=0x7f96c4198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c41013b0 0x7f96c4198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 -- 172.21.15.78:0/1657183488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c419f710 con 0x7f96c4101cc0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c2d9d640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c4101cc0 0x7f96c419f0a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00efd0 tx=0x7f96bc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96b37fe640 1 -- 172.21.15.78:0/1657183488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc003b70 con 0x7f96c4101cc0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96b37fe640 1 -- 172.21.15.78:0/1657183488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96bc003d10 con 0x7f96c4101cc0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c419fa00 con 0x7f96c4101cc0 2024-03-16T11:05:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96b37fe640 1 -- 172.21.15.78:0/1657183488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc010980 con 0x7f96c4101cc0 2024-03-16T11:05:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.576+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c419ff20 con 0x7f96c4101cc0 2024-03-16T11:05:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.577+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c41a01b0 con 0x7f96c4101cc0 2024-03-16T11:05:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.579+0000 7f96b37fe640 1 -- 172.21.15.78:0/1657183488 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f96bc004020 con 0x7f96c4101cc0 2024-03-16T11:05:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.579+0000 7f96b37fe640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.579+0000 7f96c259c640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.579+0000 7f96b37fe640 1 -- 172.21.15.78:0/1657183488 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f96bc0cefd0 con 0x7f96c4101cc0 2024-03-16T11:05:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.580+0000 7f96c259c640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f96c4069bb0 tx=0x7f96b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.584+0000 7f96b37fe640 1 -- 172.21.15.78:0/1657183488 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96bc09bf10 con 0x7f96c4101cc0 2024-03-16T11:05:59.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.794+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96c41a0380 con 0x7f96c4101cc0 2024-03-16T11:05:59.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.794+0000 7f96b37fe640 1 -- 172.21.15.78:0/1657183488 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7f96bc09b810 con 0x7f96c4101cc0 2024-03-16T11:05:59.795 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:05:59.795 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:05:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:05:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96ac0a2910 msgr2=0x7f96ac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f96c4069bb0 tx=0x7f96b4005f90 comp rx=0 tx=0).stop 2024-03-16T11:05:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c4101cc0 msgr2=0x7f96c419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c4101cc0 0x7f96c419f0a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00efd0 tx=0x7f96bc00c6a0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 shutdown_connections 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c4101cc0 0x7f96c419f0a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c41013b0 0x7f96c4198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 --2- 172.21.15.78:0/1657183488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96c41009f0 0x7f96c4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.796+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 >> 172.21.15.78:0/1657183488 conn(0x7f96c40fb420 msgr2=0x7f96c40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.797+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 shutdown_connections 2024-03-16T11:05:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.797+0000 7f96c8edc640 1 -- 172.21.15.78:0/1657183488 wait complete. 2024-03-16T11:05:59.808 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 -- 172.21.15.78:0/1435336763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1033a0 msgr2=0x7f516c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 --2- 172.21.15.78:0/1435336763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1033a0 0x7f516c109c30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f515c009ca0 tx=0x7f515c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 -- 172.21.15.78:0/1435336763 shutdown_connections 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 --2- 172.21.15.78:0/1435336763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1033a0 0x7f516c109c30 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 --2- 172.21.15.78:0/1435336763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f516c1029e0 0x7f516c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 --2- 172.21.15.78:0/1435336763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f516c1017f0 0x7f516c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 -- 172.21.15.78:0/1435336763 >> 172.21.15.78:0/1435336763 conn(0x7f516c0fcdc0 msgr2=0x7f516c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 -- 172.21.15.78:0/1435336763 shutdown_connections 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.910+0000 7f5170a19640 1 -- 172.21.15.78:0/1435336763 wait complete. 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 Processor -- start 2024-03-16T11:05:59.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 -- start start 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f516c1017f0 0x7f516c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1029e0 0x7f516c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f516c1033a0 0x7f516c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f516c10d320 con 0x7f516c1029e0 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f516c10d1a0 con 0x7f516c1017f0 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f516c10d4a0 con 0x7f516c1033a0 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1029e0 0x7f516c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1029e0 0x7f516c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54876/0 (socket says 172.21.15.78:54876) 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 -- 172.21.15.78:0/3145082471 learned_addr learned my addr 172.21.15.78:0/3145082471 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f516a59c640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f516c1017f0 0x7f516c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f516ad9d640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f516c1033a0 0x7f516c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 -- 172.21.15.78:0/3145082471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f516c1033a0 msgr2=0x7f516c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f516c1033a0 0x7f516c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 -- 172.21.15.78:0/3145082471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f516c1017f0 msgr2=0x7f516c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f516c1017f0 0x7f516c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 -- 172.21.15.78:0/3145082471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c19f640 con 0x7f516c1029e0 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5169d9b640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1029e0 0x7f516c197f60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f516000e9f0 tx=0x7f516000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f515b7fe640 1 -- 172.21.15.78:0/3145082471 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516000cd80 con 0x7f516c1029e0 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f515b7fe640 1 -- 172.21.15.78:0/3145082471 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51600054d0 con 0x7f516c1029e0 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516c19f930 con 0x7f516c1029e0 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f515b7fe640 1 -- 172.21.15.78:0/3145082471 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160010490 con 0x7f516c1029e0 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.911+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f516c19fe30 con 0x7f516c1029e0 2024-03-16T11:05:59.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.912+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516c108b30 con 0x7f516c1029e0 2024-03-16T11:05:59.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.913+0000 7f515b7fe640 1 -- 172.21.15.78:0/3145082471 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f5160005020 con 0x7f516c1029e0 2024-03-16T11:05:59.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.914+0000 7f515b7fe640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51540a2890 0x7f51540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:05:59.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.914+0000 7f515b7fe640 1 -- 172.21.15.78:0/3145082471 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f51600cef70 con 0x7f516c1029e0 2024-03-16T11:05:59.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.914+0000 7f516a59c640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51540a2890 0x7f51540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:05:59.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.914+0000 7f516a59c640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51540a2890 0x7f51540a4d50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f516c102840 tx=0x7f5164005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:05:59.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:05:59.918+0000 7f515b7fe640 1 -- 172.21.15.78:0/3145082471 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f516009beb0 con 0x7f516c1029e0 2024-03-16T11:06:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.094+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f516c02d390 con 0x7f516c1029e0 2024-03-16T11:06:00.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.095+0000 7f515b7fe640 1 -- 172.21.15.78:0/3145082471 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f516009b7b0 con 0x7f516c1029e0 2024-03-16T11:06:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:06:00.096 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:06:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.097+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51540a2890 msgr2=0x7f51540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.097+0000 7f5170a19640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51540a2890 0x7f51540a4d50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f516c102840 tx=0x7f5164005f90 comp rx=0 tx=0).stop 2024-03-16T11:06:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.097+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1029e0 msgr2=0x7f516c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.097+0000 7f5170a19640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1029e0 0x7f516c197f60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f516000e9f0 tx=0x7f516000edf0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 shutdown_connections 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f51540a2890 0x7f51540a4d50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f516c1033a0 0x7f516c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f516c1029e0 0x7f516c197f60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 --2- 172.21.15.78:0/3145082471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f516c1017f0 0x7f516c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 >> 172.21.15.78:0/3145082471 conn(0x7f516c0fcdc0 msgr2=0x7f516c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 shutdown_connections 2024-03-16T11:06:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.098+0000 7f5170a19640 1 -- 172.21.15.78:0/3145082471 wait complete. 2024-03-16T11:06:00.111 DEBUG:tasks.fwd_scrub:Yielding 2024-03-16T11:06:00.111 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-16T11:06:00.121 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T11:06:00.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-16T11:06:00.122 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-16T11:06:00.122 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T11:06:00.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:06:00.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T11:06:00.124 INFO:teuthology.run_tasks:Running task workunit... 2024-03-16T11:06:00.142 INFO:tasks.workunit:Pulling workunits from ref 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:06:00.143 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-16T11:06:00.143 INFO:tasks.workunit:timeout=3h 2024-03-16T11:06:00.143 INFO:tasks.workunit:cleanup=True 2024-03-16T11:06:00.143 DEBUG:teuthology.orchestra.run.smithi078:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-16 11:04:05.173367005 +0000 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-16 11:04:05.173367005 +0000 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-16 11:05:53.604570307 +0000 2024-03-16T11:06:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-03-16T11:06:00.160 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-16T11:06:00.160 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 -- 172.21.15.78:0/783110279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81018d0 msgr2=0x7f37e8101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 --2- 172.21.15.78:0/783110279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81018d0 0x7f37e8101d50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009900 tx=0x7f37e402bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 -- 172.21.15.78:0/783110279 shutdown_connections 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 --2- 172.21.15.78:0/783110279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e8102290 0x7f37e8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 --2- 172.21.15.78:0/783110279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81018d0 0x7f37e8101d50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 --2- 172.21.15.78:0/783110279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37e81006e0 0x7f37e8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 -- 172.21.15.78:0/783110279 >> 172.21.15.78:0/783110279 conn(0x7f37e80fbc70 msgr2=0x7f37e80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 -- 172.21.15.78:0/783110279 shutdown_connections 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.230+0000 7f37efce8640 1 -- 172.21.15.78:0/783110279 wait complete. 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 Processor -- start 2024-03-16T11:06:00.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 -- start start 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81006e0 0x7f37e8196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37e81018d0 0x7f37e8196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e8102290 0x7f37e819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f37e810c210 con 0x7f37e81006e0 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e810c090 con 0x7f37e8102290 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37efce8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37e810c390 con 0x7f37e81018d0 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81006e0 0x7f37e8196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81006e0 0x7f37e8196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54896/0 (socket says 172.21.15.78:54896) 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 -- 172.21.15.78:0/358257665 learned_addr learned my addr 172.21.15.78:0/358257665 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37ee285640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e8102290 0x7f37e819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37ed283640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37e81018d0 0x7f37e8196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.235 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:06:00.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 -- 172.21.15.78:0/358257665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37e81018d0 msgr2=0x7f37e8196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37e81018d0 0x7f37e8196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 -- 172.21.15.78:0/358257665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e8102290 msgr2=0x7f37e819df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e8102290 0x7f37e819df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 -- 172.21.15.78:0/358257665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e819e570 con 0x7f37e81006e0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.231+0000 7f37eda84640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81006e0 0x7f37e8196950 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f37d800e9c0 tx=0x7f37d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.232+0000 7f37deffd640 1 -- 172.21.15.78:0/358257665 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d800cd80 con 0x7f37e81006e0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.232+0000 7f37deffd640 1 -- 172.21.15.78:0/358257665 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d80035f0 con 0x7f37e81006e0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.232+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e819e860 con 0x7f37e81006e0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.232+0000 7f37deffd640 1 -- 172.21.15.78:0/358257665 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8010690 con 0x7f37e81006e0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.232+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e819ed30 con 0x7f37e81006e0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.234+0000 7f37deffd640 1 -- 172.21.15.78:0/358257665 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f37d80037a0 con 0x7f37e81006e0 2024-03-16T11:06:00.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.234+0000 7f37f29ac740 1 Processor -- start 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37f29ac740 1 -- start start 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37deffd640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f37d40a2890 0x7f37d40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e2c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e31290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e33d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37f29ac740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557840d8c0d0 con 0x557840e2bda0 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37f29ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557840d34c70 con 0x557840e2c790 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37f29ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557840d88170 con 0x557840e31900 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37ee285640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f37d40a2890 0x7f37d40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e2c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37deffd640 1 -- 172.21.15.78:0/358257665 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f37d80cf140 con 0x7f37e81006e0 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e2c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54908/0 (socket says 172.21.15.78:54908) 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 -- 172.21.15.78:0/653201872 learned_addr learned my addr 172.21.15.78:0/653201872 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37ed283640 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e31290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37ee285640 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e33d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 -- 172.21.15.78:0/653201872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 msgr2=0x557840e33d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e33d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 -- 172.21.15.78:0/653201872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 msgr2=0x557840e31290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e31290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 -- 172.21.15.78:0/653201872 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557840b687e0 con 0x557840e2bda0 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37ee285640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f37d40a2890 0x7f37d40a4d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f37e0005fd0 tx=0x7f37e0009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.235+0000 7f37eda84640 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e2c1c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f37d800cfd0 tx=0x7f37d8097d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9963333d19e76f83 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37d3fff640 1 -- 172.21.15.78:0/653201872 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d809c080 con 0x557840e2bda0 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37d3fff640 1 -- 172.21.15.78:0/653201872 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d809b980 con 0x557840e2bda0 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37d3fff640 1 -- 172.21.15.78:0/653201872 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8097ff0 con 0x557840e2bda0 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 -- 172.21.15.78:0/653201872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 msgr2=0x557840e2c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e2c1c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f37d800cfd0 tx=0x7f37d8097d50 comp rx=0 tx=0).stop 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 -- 172.21.15.78:0/653201872 shutdown_connections 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e33d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e31290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 --2- 172.21.15.78:0/653201872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e2c1c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 -- 172.21.15.78:0/653201872 >> 172.21.15.78:0/653201872 conn(0x557840defc50 msgr2=0x557840e23300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 -- 172.21.15.78:0/653201872 shutdown_connections 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.236+0000 7f37f29ac740 1 -- 172.21.15.78:0/653201872 wait complete. 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 -- start start 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 Processor -- start 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2bda0 0x557840e362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2c790 0x557840e36800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557840e348b0 con 0x557840e31900 2024-03-16T11:06:00.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557840d480d0 con 0x557840e2bda0 2024-03-16T11:06:00.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557840e34a30 con 0x557840e2c790 2024-03-16T11:06:00.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37f29ac740 10 client.0 _ll_register_callbacks cb 0x557840d11760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:00.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37ee285640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37ee285640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54910/0 (socket says 172.21.15.78:54910) 2024-03-16T11:06:00.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37ee285640 1 -- 172.21.15.78:0/2470703465 learned_addr learned my addr 172.21.15.78:0/2470703465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37eda84640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2bda0 0x557840e362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.237+0000 7f37ed283640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2c790 0x557840e36800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37ee285640 1 -- 172.21.15.78:0/2470703465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2c790 msgr2=0x557840e36800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37ee285640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2c790 0x557840e36800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37ee285640 1 -- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2bda0 msgr2=0x557840e362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37ee285640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2bda0 0x557840e362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37ee285640 1 -- 172.21.15.78:0/2470703465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557840bc8f70 con 0x557840e31900 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37eda84640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2bda0 0x557840e362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37ee285640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3d840 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f37e000fcf0 tx=0x7f37e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253cc4aa5c3480a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37d17fa640 1 -- 172.21.15.78:0/2470703465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0004b20 con 0x557840e31900 2024-03-16T11:06:00.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37d17fa640 1 -- 172.21.15.78:0/2470703465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f37e0004cc0 con 0x557840e31900 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d809b980 con 0x557840e31900 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37d17fa640 1 -- 172.21.15.78:0/2470703465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f37e0005020 con 0x557840e31900 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 10 client.15429 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557840a60170 con 0x557840e31900 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37d17fa640 1 -- 172.21.15.78:0/2470703465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d809b980 con 0x557840e31900 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37d17fa640 1 -- 172.21.15.78:0/2470703465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f37e001f070 con 0x557840e31900 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 10 client.15429 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 10 client.15429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 20 client.15429 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840e41d90 0x557840e44230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.238+0000 7f37f29ac740 4 client.15429 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:00.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.239+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557840d99610 con 0x557840e41d90 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.239+0000 7f37eda84640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840e41d90 0x557840e44230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.239+0000 7f37eda84640 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840e41d90 0x557840e44230 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723de16a303e2411 server_cookie=eb4a5ca890ecc5b1 in_seq=0 out_seq=0 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.239+0000 7f37d17fa640 10 client.15429 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37d17fa640 1 -- 172.21.15.78:0/2470703465 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557840d99610 con 0x557840e41d90 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37d17fa640 10 client.15429 handle_command_reply: tid=0 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 client.15429 shutdown 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 msgr2=0x557840e3d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3d840 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f37e000fcf0 tx=0x7f37e0004ab0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 shutdown_connections 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840e41d90 0x557840e44230 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3d840 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2c790 0x557840e36800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 --2- 172.21.15.78:0/2470703465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2bda0 0x557840e362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37d17fa640 0 client.15429 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 >> 172.21.15.78:0/2470703465 conn(0x557840defc50 msgr2=0x557840e224e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 shutdown_connections 2024-03-16T11:06:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 1 -- 172.21.15.78:0/2470703465 wait complete. 2024-03-16T11:06:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.240+0000 7f37f29ac740 20 client.15429 trim_cache size 0 max 16384 2024-03-16T11:06:00.252 INFO:tasks.workunit.client.0.smithi078.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-16T11:06:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 Processor -- start 2024-03-16T11:06:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 -- start start 2024-03-16T11:06:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e21c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e22190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a9290 con 0x557840e2bda0 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a6b60 con 0x557840e2c790 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37f29ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a4990 con 0x557840e31900 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37eda84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e21c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37eda84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e21c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54920/0 (socket says 172.21.15.78:54920) 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37eda84640 1 -- 172.21.15.78:0/3071916842 learned_addr learned my addr 172.21.15.78:0/3071916842 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37ed283640 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.260+0000 7f37ee285640 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e22190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37eda84640 1 -- 172.21.15.78:0/3071916842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 msgr2=0x557840e22190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37eda84640 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e22190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37eda84640 1 -- 172.21.15.78:0/3071916842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 msgr2=0x557840e3d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37eda84640 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37eda84640 1 -- 172.21.15.78:0/3071916842 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557840e1ed40 con 0x557840e2bda0 2024-03-16T11:06:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37ed283640 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37eda84640 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e21c50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f37d809b910 tx=0x7f37d8097c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eceef935c69e2f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37d37fe640 1 -- 172.21.15.78:0/3071916842 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d809b980 con 0x557840e2bda0 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37d37fe640 1 -- 172.21.15.78:0/3071916842 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d809c080 con 0x557840e2bda0 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.261+0000 7f37d37fe640 1 -- 172.21.15.78:0/3071916842 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d80976d0 con 0x557840e2bda0 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- 172.21.15.78:0/3071916842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 msgr2=0x557840e21c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e21c50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f37d809b910 tx=0x7f37d8097c20 comp rx=0 tx=0).stop 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- 172.21.15.78:0/3071916842 shutdown_connections 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e31900 0x557840e22190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 --2- 172.21.15.78:0/3071916842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e2bda0 0x557840e21c50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- 172.21.15.78:0/3071916842 >> 172.21.15.78:0/3071916842 conn(0x557840defc50 msgr2=0x557840e34dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- 172.21.15.78:0/3071916842 shutdown_connections 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- 172.21.15.78:0/3071916842 wait complete. 2024-03-16T11:06:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- start start 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 Processor -- start 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2bda0 0x557840e3b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a5020 con 0x557840e31900 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a6490 con 0x557840e2c790 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.262+0000 7f37f29ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a56f0 con 0x557840e2bda0 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37f29ac740 10 client.0 _ll_register_callbacks cb 0x7f37e0013dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54930/0 (socket says 172.21.15.78:54930) 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 -- 172.21.15.78:0/1032395389 learned_addr learned my addr 172.21.15.78:0/1032395389 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ed283640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37eda84640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2bda0 0x557840e3b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 -- 172.21.15.78:0/1032395389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2bda0 msgr2=0x557840e3b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2bda0 0x557840e3b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 -- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 msgr2=0x557840e3b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 -- 172.21.15.78:0/1032395389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557840e3cb90 con 0x557840e31900 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ed283640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37ee285640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3c5a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f37e000fcc0 tx=0x7f37e0003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ff9ed7a230afc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37d17fa640 1 -- 172.21.15.78:0/1032395389 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0011ca0 con 0x557840e31900 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37d17fa640 1 -- 172.21.15.78:0/1032395389 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f37e002a900 con 0x557840e31900 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d809c080 con 0x557840e31900 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37d17fa640 1 -- 172.21.15.78:0/1032395389 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f37e0003e10 con 0x557840e31900 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37f29ac740 10 client.15435 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.263+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557840ea82e0 con 0x557840e31900 2024-03-16T11:06:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37d17fa640 1 -- 172.21.15.78:0/1032395389 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d809c080 con 0x557840e31900 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37d17fa640 1 -- 172.21.15.78:0/1032395389 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f37e0046050 con 0x557840e31900 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37f29ac740 10 client.15435 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37f29ac740 10 client.15435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37f29ac740 20 client.15435 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37f29ac740 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840ea8c50 0x557840eab070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37f29ac740 4 client.15435 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557840d93960 con 0x557840ea8c50 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37eda84640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840ea8c50 0x557840eab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37d17fa640 10 client.15435 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.264+0000 7f37eda84640 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840ea8c50 0x557840eab070 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a31eac91d935745 server_cookie=5a78df0a81143bef in_seq=0 out_seq=0 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.265+0000 7f37d17fa640 1 -- 172.21.15.78:0/1032395389 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557840d93960 con 0x557840ea8c50 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.265+0000 7f37d17fa640 10 client.15435 handle_command_reply: tid=0 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.265+0000 7f37f29ac740 1 client.15435 shutdown 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.265+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 msgr2=0x557840e3c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.265+0000 7f37f29ac740 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3c5a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f37e000fcc0 tx=0x7f37e0003c20 comp rx=0 tx=0).stop 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 shutdown_connections 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557840ea8c50 0x557840eab070 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557840e31900 0x557840e3c5a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557840e2c790 0x557840e3b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37d17fa640 0 client.15435 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 --2- 172.21.15.78:0/1032395389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557840e2bda0 0x557840e3b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 >> 172.21.15.78:0/1032395389 conn(0x557840defc50 msgr2=0x557840e26540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 shutdown_connections 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 1 -- 172.21.15.78:0/1032395389 wait complete. 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37f29ac740 20 client.15435 trim_cache size 0 max 16384 2024-03-16T11:06:00.272 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:06:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-16T11:06:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "1b30c3bb-e6a9-4052-9249-3e6cd893bca4", 2024-03-16T11:06:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-16T11:06:00.273 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f37d40a2890 msgr2=0x7f37d40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f37d40a2890 0x7f37d40a4d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f37e0005fd0 tx=0x7f37e0009b80 comp rx=0 tx=0).stop 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81006e0 msgr2=0x7f37e8196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81006e0 0x7f37e8196950 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f37d800e9c0 tx=0x7f37d800ee80 comp rx=0 tx=0).stop 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 shutdown_connections 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37e8102290 0x7f37e819df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f37d40a2890 0x7f37d40a4d50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37e81018d0 0x7f37e8196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 --2- 172.21.15.78:0/358257665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e81006e0 0x7f37e8196950 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 >> 172.21.15.78:0/358257665 conn(0x7f37e80fbc70 msgr2=0x7f37e8104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 shutdown_connections 2024-03-16T11:06:00.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.266+0000 7f37efce8640 1 -- 172.21.15.78:0/358257665 wait complete. 2024-03-16T11:06:00.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:06:00.278 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:06:00.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.427+0000 7f9ebb59e640 1 -- 172.21.15.78:0/3739327483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc102140 msgr2=0x7f9ebc104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.427+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/3739327483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc102140 0x7f9ebc104530 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009900 tx=0x7f9eb402bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 -- 172.21.15.78:0/3739327483 shutdown_connections 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/3739327483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ebc1073a0 0x7f9ebc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/3739327483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc104a70 0x7f9ebc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/3739327483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc102140 0x7f9ebc104530 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 -- 172.21.15.78:0/3739327483 >> 172.21.15.78:0/3739327483 conn(0x7f9ebc0fbb50 msgr2=0x7f9ebc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 -- 172.21.15.78:0/3739327483 shutdown_connections 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 -- 172.21.15.78:0/3739327483 wait complete. 2024-03-16T11:06:00.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 Processor -- start 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.428+0000 7f9ebb59e640 1 -- start start 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ebc102140 0x7f9ebc197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc104a70 0x7f9ebc197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc1073a0 0x7f9ebc19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc10d210 con 0x7f9ebc104a70 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc10d090 con 0x7f9ebc102140 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc10d390 con 0x7f9ebc1073a0 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc104a70 0x7f9ebc197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc104a70 0x7f9ebc197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54954/0 (socket says 172.21.15.78:54954) 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eba59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ebc102140 0x7f9ebc197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/538755974 learned_addr learned my addr 172.21.15.78:0/538755974 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc1073a0 0x7f9ebc19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/538755974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc1073a0 msgr2=0x7f9ebc19eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc1073a0 0x7f9ebc19eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/538755974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ebc102140 msgr2=0x7f9ebc197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ebc102140 0x7f9ebc197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/538755974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ebc19f540 con 0x7f9ebc104a70 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eba59c640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ebc102140 0x7f9ebc197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc104a70 0x7f9ebc197e60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400e9f0 tx=0x7f9ea400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb37fe640 1 -- 172.21.15.78:0/538755974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea400ccc0 con 0x7f9ebc104a70 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ebc19f830 con 0x7f9ebc104a70 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb37fe640 1 -- 172.21.15.78:0/538755974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ea40054d0 con 0x7f9ebc104a70 2024-03-16T11:06:00.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9eb37fe640 1 -- 172.21.15.78:0/538755974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4010430 con 0x7f9ebc104a70 2024-03-16T11:06:00.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.429+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ebc19fd30 con 0x7f9ebc104a70 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.431+0000 7f9eb37fe640 1 -- 172.21.15.78:0/538755974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f9ea4005020 con 0x7f9ebc104a70 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 Processor -- start 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 -- start start 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa611760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa616830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa619330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5564fa576500 con 0x5564fa611d30 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5564fa51ebe0 con 0x5564fa616ea0 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ec35df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5564fa5725a0 con 0x5564fa611340 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9eb37fe640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9ea80a2890 0x7f9ea80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9eba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa616830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9eba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa616830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54966/0 (socket says 172.21.15.78:54966) 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9eba59c640 1 -- 172.21.15.78:0/3972468680 learned_addr learned my addr 172.21.15.78:0/3972468680 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa619330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa611760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.432+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9ea80a2890 0x7f9ea80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eb37fe640 1 -- 172.21.15.78:0/538755974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9ea40cef80 con 0x7f9ebc104a70 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eba59c640 1 -- 172.21.15.78:0/3972468680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 msgr2=0x5564fa611760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eba59c640 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa611760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eba59c640 1 -- 172.21.15.78:0/3972468680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 msgr2=0x5564fa619330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eba59c640 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa619330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eba59c640 1 -- 172.21.15.78:0/3972468680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564fa5324b0 con 0x5564fa611d30 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa619330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eba59c640 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa616830 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009900 tx=0x7f9eb40095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa0f157ce099ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9ea80a2890 0x7f9ea80a4d50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5564fa612590 tx=0x7f9eac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eb0ff9640 1 -- 172.21.15.78:0/3972468680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4045020 con 0x5564fa611d30 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eb0ff9640 1 -- 172.21.15.78:0/3972468680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9eb402cc90 con 0x5564fa611d30 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.433+0000 7f9eb0ff9640 1 -- 172.21.15.78:0/3972468680 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4004920 con 0x5564fa611d30 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 -- 172.21.15.78:0/3972468680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 msgr2=0x5564fa616830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa616830 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009900 tx=0x7f9eb40095f0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 -- 172.21.15.78:0/3972468680 shutdown_connections 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa619330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa616830 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3972468680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa611760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 -- 172.21.15.78:0/3972468680 >> 172.21.15.78:0/3972468680 conn(0x5564fa60c8b0 msgr2=0x5564fa60ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 -- 172.21.15.78:0/3972468680 shutdown_connections 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 -- 172.21.15.78:0/3972468680 wait complete. 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 -- start start 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 Processor -- start 2024-03-16T11:06:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.434+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa61b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa61bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa622d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5564fa619df0 con 0x5564fa611d30 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5564fa532160 con 0x5564fa616ea0 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5564fa619f70 con 0x5564fa611340 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 10 client.0 _ll_register_callbacks cb 0x5564fa516150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa61bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa61bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54968/0 (socket says 172.21.15.78:54968) 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/1131720701 learned_addr learned my addr 172.21.15.78:0/1131720701 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eba59c640 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa61b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa622d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/1131720701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 msgr2=0x5564fa61b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa61b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 msgr2=0x5564fa622d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa622d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 -- 172.21.15.78:0/1131720701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564fa3547d0 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa61bdb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4098190 tx=0x7f9ea401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50571ec0cdecffcc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ea1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/1131720701 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40046c0 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/1131720701 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9ea40d10d0 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eb402cc90 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/1131720701 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9ea4005310 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 10 client.15447 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.435+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5564fa28a3f0 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/1131720701 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9eb402cc90 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/1131720701 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f9ea40a3ad0 con 0x5564fa611d30 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ec35df740 10 client.15447 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ec35df740 10 client.15447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:00.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ec35df740 20 client.15447 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ec35df740 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa627370 0x5564fa6297a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ec35df740 4 client.15447 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564fa583a40 con 0x5564fa627370 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9eba59c640 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa627370 0x5564fa6297a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9eba59c640 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa627370 0x5564fa6297a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b457d0c091379f52 server_cookie=4c9d71dab6c70dd8 in_seq=0 out_seq=0 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.436+0000 7f9ea1ffb640 10 client.15447 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/1131720701 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5564fa583a40 con 0x5564fa627370 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ea1ffb640 10 client.15447 handle_command_reply: tid=0 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 client.15447 shutdown 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 msgr2=0x5564fa61bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa61bdb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4098190 tx=0x7f9ea401d000 comp rx=0 tx=0).stop 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 shutdown_connections 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa627370 0x5564fa6297a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa616ea0 0x5564fa622d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611d30 0x5564fa61bdb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ea1ffb640 0 client.15447 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 --2- 172.21.15.78:0/1131720701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611340 0x5564fa61b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 >> 172.21.15.78:0/1131720701 conn(0x5564fa60c8b0 msgr2=0x5564fa60ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 shutdown_connections 2024-03-16T11:06:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 1 -- 172.21.15.78:0/1131720701 wait complete. 2024-03-16T11:06:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.437+0000 7f9ec35df740 20 client.15447 trim_cache size 0 max 16384 2024-03-16T11:06:00.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.454+0000 7f9ec35df740 1 Processor -- start 2024-03-16T11:06:00.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.454+0000 7f9ec35df740 1 -- start start 2024-03-16T11:06:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611340 0x5564fa6078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611d30 0x5564fa622d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa616ea0 0x5564fa60b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ec35df740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc1ab210 con 0x5564fa611340 2024-03-16T11:06:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ec35df740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc1a8ae0 con 0x5564fa611d30 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ec35df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc1a6910 con 0x5564fa616ea0 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611340 0x5564fa6078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611340 0x5564fa6078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54980/0 (socket says 172.21.15.78:54980) 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 -- 172.21.15.78:0/3440774156 learned_addr learned my addr 172.21.15.78:0/3440774156 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611d30 0x5564fa622d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa616ea0 0x5564fa60b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 -- 172.21.15.78:0/3440774156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa616ea0 msgr2=0x5564fa60b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa616ea0 0x5564fa60b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 -- 172.21.15.78:0/3440774156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611d30 msgr2=0x5564fa622d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611d30 0x5564fa622d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 -- 172.21.15.78:0/3440774156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564fa61aa20 con 0x5564fa611340 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9eba59c640 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611340 0x5564fa6078f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4005ec0 tx=0x7f9eb4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67a4c86d485fc64 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ea3fff640 1 -- 172.21.15.78:0/3440774156 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4050070 con 0x5564fa611340 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ea3fff640 1 -- 172.21.15.78:0/3440774156 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9eb402cc90 con 0x5564fa611340 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.455+0000 7f9ea3fff640 1 -- 172.21.15.78:0/3440774156 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4045040 con 0x5564fa611340 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- 172.21.15.78:0/3440774156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611340 msgr2=0x5564fa6078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611340 0x5564fa6078f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4005ec0 tx=0x7f9eb4000bc0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- 172.21.15.78:0/3440774156 shutdown_connections 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa616ea0 0x5564fa60b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611d30 0x5564fa622d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 --2- 172.21.15.78:0/3440774156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa611340 0x5564fa6078f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- 172.21.15.78:0/3440774156 >> 172.21.15.78:0/3440774156 conn(0x5564fa60c8b0 msgr2=0x5564fa6132c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- 172.21.15.78:0/3440774156 shutdown_connections 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- 172.21.15.78:0/3440774156 wait complete. 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- start start 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 Processor -- start 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611340 0x5564fa620370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611d30 0x5564fa6208b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa616ea0 0x5564fa621850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc1a6fa0 con 0x5564fa616ea0 2024-03-16T11:06:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc1a8410 con 0x5564fa611340 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc1a7670 con 0x5564fa611d30 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.456+0000 7f9ec35df740 10 client.0 _ll_register_callbacks cb 0x7f9ea400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa616ea0 0x5564fa621850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa616ea0 0x5564fa621850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54984/0 (socket says 172.21.15.78:54984) 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 -- 172.21.15.78:0/929999963 learned_addr learned my addr 172.21.15.78:0/929999963 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9eba59c640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611340 0x5564fa620370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611d30 0x5564fa6208b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 -- 172.21.15.78:0/929999963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611d30 msgr2=0x5564fa6208b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611d30 0x5564fa6208b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 -- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611340 msgr2=0x5564fa620370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611340 0x5564fa620370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 -- 172.21.15.78:0/929999963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564fa24a0f0 con 0x5564fa616ea0 2024-03-16T11:06:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9eb9d9b640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611d30 0x5564fa6208b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ebad9d640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa616ea0 0x5564fa621850 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9eac003ba0 tx=0x7f9eac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8479bd0696ed4acf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ea1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/929999963 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac02e060 con 0x5564fa616ea0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/929999963 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9eac020450 con 0x5564fa616ea0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eb402cc90 con 0x5564fa616ea0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/929999963 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9eac020650 con 0x5564fa616ea0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ec35df740 10 client.15453 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5564fa53d9d0 con 0x5564fa616ea0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.457+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/929999963 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9eb402cc90 con 0x5564fa616ea0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/929999963 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f9eac033070 con 0x5564fa616ea0 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 10 client.15453 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 10 client.15453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 20 client.15453 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa68dca0 0x5564fa6900c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 4 client.15453 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5564fa57dd90 con 0x5564fa68dca0 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9eba59c640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa68dca0 0x5564fa6900c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ea1ffb640 10 client.15453 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9eba59c640 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa68dca0 0x5564fa6900c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bf0b0f6be21eab6 server_cookie=8792fdbf685b3f0 in_seq=0 out_seq=0 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ea1ffb640 1 -- 172.21.15.78:0/929999963 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5564fa57dd90 con 0x5564fa68dca0 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ea1ffb640 10 client.15453 handle_command_reply: tid=0 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 client.15453 shutdown 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa616ea0 msgr2=0x5564fa621850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa616ea0 0x5564fa621850 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9eac003ba0 tx=0x7f9eac004a40 comp rx=0 tx=0).stop 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 shutdown_connections 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5564fa68dca0 0x5564fa6900c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564fa616ea0 0x5564fa621850 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ea1ffb640 0 client.15453 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564fa611d30 0x5564fa6208b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 --2- 172.21.15.78:0/929999963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564fa611340 0x5564fa620370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.458+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 >> 172.21.15.78:0/929999963 conn(0x5564fa60c8b0 msgr2=0x5564fa60b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 shutdown_connections 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ec35df740 1 -- 172.21.15.78:0/929999963 wait complete. 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ec35df740 20 client.15453 trim_cache size 0 max 16384 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T11:06:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-16T11:06:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "1b30c3bb-e6a9-4052-9249-3e6cd893bca4": { 2024-03-16T11:06:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-16T11:06:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "1b30c3bb-e6a9-4052-9249-3e6cd893bca4", 2024-03-16T11:06:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-16T11:06:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:06:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:06:00.470 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9ea80a2890 msgr2=0x7f9ea80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9ea80a2890 0x7f9ea80a4d50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5564fa612590 tx=0x7f9eac009500 comp rx=0 tx=0).stop 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc104a70 msgr2=0x7f9ebc197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc104a70 0x7f9ebc197e60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400e9f0 tx=0x7f9ea400eeb0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 shutdown_connections 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f9ea80a2890 0x7f9ea80a4d50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc1073a0 0x7f9ebc19eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ebc104a70 0x7f9ebc197e60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 --2- 172.21.15.78:0/538755974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ebc102140 0x7f9ebc197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:00.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 >> 172.21.15.78:0/538755974 conn(0x7f9ebc0fbb50 msgr2=0x7f9ebc102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:00.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 shutdown_connections 2024-03-16T11:06:00.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:00.459+0000 7f9ebb59e640 1 -- 172.21.15.78:0/538755974 wait complete. 2024-03-16T11:06:00.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:00 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2460923216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:06:00.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:00 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4168030053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:06:00.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:00 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1657183488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:06:00.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:00 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3145082471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:06:00.473 INFO:tasks.cephfs.filesystem:scrub status for tag:1b30c3bb-e6a9-4052-9249-3e6cd893bca4 - {'path': '/', 'tag': '1b30c3bb-e6a9-4052-9249-3e6cd893bca4', 'options': 'recursive,force'} 2024-03-16T11:06:00.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:00 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2460923216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:06:00.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:00 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4168030053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:06:00.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:00 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1657183488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:06:00.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:00 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3145082471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:06:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:00 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2460923216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:06:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:00 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4168030053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:06:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:00 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1657183488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:06:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:00 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3145082471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:06:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:01 smithi098 ceph-mon[35848]: pgmap v349: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-16T11:06:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:01 smithi098 ceph-mon[35848]: scrub queued for path: / 2024-03-16T11:06:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:01 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:01 smithi098 ceph-mon[35848]: scrub summary: active paths [/] 2024-03-16T11:06:01.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:01 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:01 smithi153 ceph-mon[35521]: pgmap v349: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-16T11:06:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:01 smithi153 ceph-mon[35521]: scrub queued for path: / 2024-03-16T11:06:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:01 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:01 smithi153 ceph-mon[35521]: scrub summary: active paths [/] 2024-03-16T11:06:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:01 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:01 smithi078 ceph-mon[32235]: pgmap v349: 97 pgs: 97 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-16T11:06:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:01 smithi078 ceph-mon[32235]: scrub queued for path: / 2024-03-16T11:06:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:01 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:01 smithi078 ceph-mon[32235]: scrub summary: active paths [/] 2024-03-16T11:06:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:01 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:03 smithi098 ceph-mon[35848]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-16T11:06:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:03 smithi153 ceph-mon[35521]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-16T11:06:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:03 smithi078 ceph-mon[32235]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-16T11:06:05.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:05 smithi098 ceph-mon[35848]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-16T11:06:05.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:05 smithi098 ceph-mon[35848]: scrub summary: idle 2024-03-16T11:06:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:05 smithi153 ceph-mon[35521]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-16T11:06:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:05 smithi153 ceph-mon[35521]: scrub summary: idle 2024-03-16T11:06:05.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:05 smithi078 ceph-mon[32235]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-16T11:06:05.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:05 smithi078 ceph-mon[32235]: scrub summary: idle 2024-03-16T11:06:07.361 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:07 smithi078 ceph-mon[32235]: pgmap v352: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:06:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:07 smithi098 ceph-mon[35848]: pgmap v352: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:06:07.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:07 smithi153 ceph-mon[35521]: pgmap v352: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:06:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:09 smithi098 ceph-mon[35848]: pgmap v353: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T11:06:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:09 smithi153 ceph-mon[35521]: pgmap v353: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T11:06:09.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:09 smithi078 ceph-mon[32235]: pgmap v353: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T11:06:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:11 smithi098 ceph-mon[35848]: pgmap v354: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-16T11:06:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:11 smithi153 ceph-mon[35521]: pgmap v354: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-16T11:06:11.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:11 smithi078 ceph-mon[32235]: pgmap v354: 97 pgs: 97 active+clean; 654 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.254+0000 7f7dda8e8640 1 -- 172.21.15.78:0/2221619332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 msgr2=0x7f7dd4104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.254+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/2221619332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd4104670 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009900 tx=0x7f7dc402bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- 172.21.15.78:0/2221619332 shutdown_connections 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/2221619332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd41074e0 0x7f7dd4109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/2221619332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dd4104bb0 0x7f7dd4106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/2221619332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd4104670 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- 172.21.15.78:0/2221619332 >> 172.21.15.78:0/2221619332 conn(0x7f7dd40fbc70 msgr2=0x7f7dd40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:13.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- 172.21.15.78:0/2221619332 shutdown_connections 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- 172.21.15.78:0/2221619332 wait complete. 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 Processor -- start 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- start start 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd41979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd4104bb0 0x7f7dd4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dd41074e0 0x7f7dd419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd410d330 con 0x7f7dd4102280 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd410d1b0 con 0x7f7dd41074e0 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.255+0000 7f7dda8e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd410d4b0 con 0x7f7dd4104bb0 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd41979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd41979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60128/0 (socket says 172.21.15.78:60128) 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 -- 172.21.15.78:0/3376059004 learned_addr learned my addr 172.21.15.78:0/3376059004 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd37fe640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd4104bb0 0x7f7dd4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dd41074e0 0x7f7dd419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 -- 172.21.15.78:0/3376059004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd4104bb0 msgr2=0x7f7dd4197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd4104bb0 0x7f7dd4197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 -- 172.21.15.78:0/3376059004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dd41074e0 msgr2=0x7f7dd419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dd41074e0 0x7f7dd419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 -- 172.21.15.78:0/3376059004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd419f5d0 con 0x7f7dd4102280 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd41979b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4006010 tx=0x7f7dc4005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3376059004 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4043020 con 0x7f7dd4102280 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3376059004 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dc40043d0 con 0x7f7dd4102280 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3376059004 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4003b80 con 0x7f7dd4102280 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd419f860 con 0x7f7dd4102280 2024-03-16T11:06:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.256+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd419fdb0 con 0x7f7dd4102280 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.258+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3376059004 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f7dc4046020 con 0x7f7dd4102280 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.258+0000 7f7ddd5ae740 1 Processor -- start 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7ddd5ae740 1 -- start start 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc7109760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc710e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7111330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dcc706ea40 con 0x55dcc710eea0 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dcc7017920 con 0x55dcc7109340 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcc706aae0 con 0x55dcc7109d30 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd17fa640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7dbc0a2940 0x7f7dbc0a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc7109760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7111330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc7109760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:47380/0 (socket says 172.21.15.78:47380) 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7111330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60140/0 (socket says 172.21.15.78:60140) 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 -- 172.21.15.78:0/1898432309 learned_addr learned my addr 172.21.15.78:0/1898432309 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc710e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7dbc0a2940 0x7f7dbc0a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3376059004 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7dc40f0cf0 con 0x7f7dd4102280 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 -- 172.21.15.78:0/1898432309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 msgr2=0x55dcc710e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc710e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 -- 172.21.15.78:0/1898432309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 msgr2=0x55dcc7111330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7111330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 -- 172.21.15.78:0/1898432309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcc702f460 con 0x55dcc7109340 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd37fe640 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7dbc0a2940 0x7f7dbc0a4e00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00fc90 tx=0x7f7dcc018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.259+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc7109760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc011e00 tx=0x7f7dcc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565e86e5d65c0fad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7dba7fc640 1 -- 172.21.15.78:0/1898432309 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc022070 con 0x55dcc7109340 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7dba7fc640 1 -- 172.21.15.78:0/1898432309 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dcc00ed10 con 0x55dcc7109340 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7dba7fc640 1 -- 172.21.15.78:0/1898432309 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc0057b0 con 0x55dcc7109340 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/1898432309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 msgr2=0x55dcc7109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc7109760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc011e00 tx=0x7f7dcc00f790 comp rx=0 tx=0).stop 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/1898432309 shutdown_connections 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc710e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/1898432309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc7109760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/1898432309 >> 172.21.15.78:0/1898432309 conn(0x55dcc71048b0 msgr2=0x55dcc7104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/1898432309 shutdown_connections 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.260+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/1898432309 wait complete. 2024-03-16T11:06:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 -- start start 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 Processor -- start 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109340 0x55dcc7113680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109d30 0x55dcc7113bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc711ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dcc7111df0 con 0x55dcc710eea0 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dcc702ad70 con 0x55dcc7109d30 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcc7111f70 con 0x55dcc7109340 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7ddd5ae740 10 client.0 _ll_register_callbacks cb 0x55dcc6d82a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc711ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109340 0x55dcc7113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109d30 0x55dcc7113bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc711ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60150/0 (socket says 172.21.15.78:60150) 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 -- 172.21.15.78:0/2091163629 learned_addr learned my addr 172.21.15.78:0/2091163629 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 -- 172.21.15.78:0/2091163629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109340 msgr2=0x55dcc7113680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109340 0x55dcc7113680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 -- 172.21.15.78:0/2091163629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109d30 msgr2=0x55dcc7113bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109d30 0x55dcc7113bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd8e85640 1 -- 172.21.15.78:0/2091163629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcc6d83210 con 0x55dcc710eea0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.261+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109340 0x55dcc7113680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc711ab90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc005c00 tx=0x7f7dcc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551442ce760c590f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7db37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7db37fe640 1 -- 172.21.15.78:0/2091163629 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc02e060 con 0x55dcc710eea0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7db37fe640 1 -- 172.21.15.78:0/2091163629 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7dcc003970 con 0x55dcc710eea0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dcc00ed10 con 0x55dcc710eea0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7db37fe640 1 -- 172.21.15.78:0/2091163629 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7dcc022040 con 0x55dcc710eea0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 10 client.15465 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55dcc6e4f620 con 0x55dcc710eea0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7db37fe640 1 -- 172.21.15.78:0/2091163629 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dcc00ed10 con 0x55dcc710eea0 2024-03-16T11:06:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7db37fe640 1 -- 172.21.15.78:0/2091163629 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7dcc033070 con 0x55dcc710eea0 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 10 client.15465 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 10 client.15465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 20 client.15465 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc711f3e0 0x55dcc71217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 4 client.15465 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.262+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcc707bf80 con 0x55dcc711f3e0 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.263+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc711f3e0 0x55dcc71217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.263+0000 7f7db37fe640 10 client.15465 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.263+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc711f3e0 0x55dcc71217d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e01f6523d9d81eca server_cookie=ae79196ae705da16 in_seq=0 out_seq=0 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7db37fe640 1 -- 172.21.15.78:0/2091163629 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dcc707bf80 con 0x55dcc711f3e0 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7db37fe640 10 client.15465 handle_command_reply: tid=0 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 client.15465 shutdown 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 msgr2=0x55dcc711ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc711ab90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc005c00 tx=0x7f7dcc009500 comp rx=0 tx=0).stop 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 shutdown_connections 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc711f3e0 0x55dcc71217d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc711ab90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7db37fe640 0 client.15465 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109d30 0x55dcc7113bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2091163629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109340 0x55dcc7113680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 >> 172.21.15.78:0/2091163629 conn(0x55dcc71048b0 msgr2=0x55dcc7103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 shutdown_connections 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2091163629 wait complete. 2024-03-16T11:06:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.264+0000 7f7ddd5ae740 20 client.15465 trim_cache size 0 max 16384 2024-03-16T11:06:13.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 Processor -- start 2024-03-16T11:06:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 -- start start 2024-03-16T11:06:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc710a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc711ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7108220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41ab470 con 0x55dcc710eea0 2024-03-16T11:06:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41a8d40 con 0x55dcc7109340 2024-03-16T11:06:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.279+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41a6b70 con 0x55dcc7109d30 2024-03-16T11:06:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7108220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60162/0 (socket says 172.21.15.78:60162) 2024-03-16T11:06:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 -- 172.21.15.78:0/781346202 learned_addr learned my addr 172.21.15.78:0/781346202 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd37fe640 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc711ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc710a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 -- 172.21.15.78:0/781346202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 msgr2=0x55dcc711ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc711ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 -- 172.21.15.78:0/781346202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 msgr2=0x55dcc710a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc710a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 -- 172.21.15.78:0/781346202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcc7108f80 con 0x55dcc710eea0 2024-03-16T11:06:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7dd8e85640 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7108220 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009730 tx=0x7f7dcc005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3dbcdf4d7f6416 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7db9ffb640 1 -- 172.21.15.78:0/781346202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc039020 con 0x55dcc710eea0 2024-03-16T11:06:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7db9ffb640 1 -- 172.21.15.78:0/781346202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dcc00ec20 con 0x55dcc710eea0 2024-03-16T11:06:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.280+0000 7f7db9ffb640 1 -- 172.21.15.78:0/781346202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc02e040 con 0x55dcc710eea0 2024-03-16T11:06:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/781346202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 msgr2=0x55dcc7108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7108220 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009730 tx=0x7f7dcc005f70 comp rx=0 tx=0).stop 2024-03-16T11:06:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/781346202 shutdown_connections 2024-03-16T11:06:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc710eea0 0x55dcc7108220 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc7109d30 0x55dcc711ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/781346202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc710a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/781346202 >> 172.21.15.78:0/781346202 conn(0x55dcc71048b0 msgr2=0x55dcc7107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/781346202 shutdown_connections 2024-03-16T11:06:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/781346202 wait complete. 2024-03-16T11:06:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- start start 2024-03-16T11:06:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 Processor -- start 2024-03-16T11:06:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc711a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc7109d30 0x55dcc7118ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc710eea0 0x55dcc7119220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41a7200 con 0x55dcc7109d30 2024-03-16T11:06:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41a8670 con 0x55dcc7109340 2024-03-16T11:06:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41a78d0 con 0x55dcc710eea0 2024-03-16T11:06:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.281+0000 7f7ddd5ae740 10 client.0 _ll_register_callbacks cb 0x7f7dc80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc7109d30 0x55dcc7118ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc710eea0 0x55dcc7119220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc711a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd8e85640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc710eea0 0x55dcc7119220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54568/0 (socket says 172.21.15.78:54568) 2024-03-16T11:06:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd8e85640 1 -- 172.21.15.78:0/2473949038 learned_addr learned my addr 172.21.15.78:0/2473949038 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd37fe640 1 -- 172.21.15.78:0/2473949038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc710eea0 msgr2=0x55dcc7119220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd37fe640 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc710eea0 0x55dcc7119220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd37fe640 1 -- 172.21.15.78:0/2473949038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 msgr2=0x55dcc711a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd37fe640 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc711a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd37fe640 1 -- 172.21.15.78:0/2473949038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcc7119760 con 0x55dcc7109d30 2024-03-16T11:06:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7dd37fe640 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc7109d30 0x55dcc7118ce0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009730 tx=0x7f7dc8012c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4633df80d69f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7db37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7db37fe640 1 -- 172.21.15.78:0/2473949038 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8023060 con 0x55dcc7109d30 2024-03-16T11:06:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7db37fe640 1 -- 172.21.15.78:0/2473949038 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7dc8004250 con 0x55dcc7109d30 2024-03-16T11:06:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dcc00ec20 con 0x55dcc7109d30 2024-03-16T11:06:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7db37fe640 1 -- 172.21.15.78:0/2473949038 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7dc8004450 con 0x55dcc7109d30 2024-03-16T11:06:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7ddd5ae740 10 client.15471 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55dcc7186450 con 0x55dcc7109d30 2024-03-16T11:06:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.282+0000 7f7db37fe640 1 -- 172.21.15.78:0/2473949038 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dcc00ec20 con 0x55dcc7109d30 2024-03-16T11:06:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7db37fe640 1 -- 172.21.15.78:0/2473949038 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7dc8028070 con 0x55dcc7109d30 2024-03-16T11:06:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7ddd5ae740 10 client.15471 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7ddd5ae740 10 client.15471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7ddd5ae740 20 client.15471 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc7186e00 0x55dcc7189220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7ddd5ae740 4 client.15471 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:06:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dcc70762d0 con 0x55dcc7186e00 2024-03-16T11:06:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc7186e00 0x55dcc7189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7db37fe640 10 client.15471 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.283+0000 7f7dd3fff640 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc7186e00 0x55dcc7189220 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55c67121574236c4 server_cookie=43a94846aeec43e1 in_seq=0 out_seq=0 2024-03-16T11:06:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7db37fe640 1 -- 172.21.15.78:0/2473949038 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dcc70762d0 con 0x55dcc7186e00 2024-03-16T11:06:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7db37fe640 10 client.15471 handle_command_reply: tid=0 2024-03-16T11:06:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 client.15471 shutdown 2024-03-16T11:06:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc7109d30 msgr2=0x55dcc7118ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc7109d30 0x55dcc7118ce0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009730 tx=0x7f7dc8012c80 comp rx=0 tx=0).stop 2024-03-16T11:06:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 shutdown_connections 2024-03-16T11:06:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dcc7186e00 0x55dcc7189220 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcc710eea0 0x55dcc7119220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dcc7109d30 0x55dcc7118ce0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7db37fe640 0 client.15471 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 --2- 172.21.15.78:0/2473949038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dcc7109340 0x55dcc711a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 >> 172.21.15.78:0/2473949038 conn(0x55dcc71048b0 msgr2=0x55dcc71083a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 shutdown_connections 2024-03-16T11:06:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 1 -- 172.21.15.78:0/2473949038 wait complete. 2024-03-16T11:06:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.294+0000 7f7ddd5ae740 20 client.15471 trim_cache size 0 max 16384 2024-03-16T11:06:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:06:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:06:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:06:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:06:13.308 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7dbc0a2940 msgr2=0x7f7dbc0a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7dbc0a2940 0x7f7dbc0a4e00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00fc90 tx=0x7f7dcc018000 comp rx=0 tx=0).stop 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 msgr2=0x7f7dd41979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd41979b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4006010 tx=0x7f7dc4005e40 comp rx=0 tx=0).stop 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 shutdown_connections 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dd41074e0 0x7f7dd419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7dbc0a2940 0x7f7dbc0a4e00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dd4104bb0 0x7f7dd4197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 --2- 172.21.15.78:0/3376059004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd4102280 0x7f7dd41979b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 >> 172.21.15.78:0/3376059004 conn(0x7f7dd40fbc70 msgr2=0x7f7dd4102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 shutdown_connections 2024-03-16T11:06:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:13.295+0000 7f7dda8e8640 1 -- 172.21.15.78:0/3376059004 wait complete. 2024-03-16T11:06:13.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:13 smithi078 ceph-mon[32235]: pgmap v355: 97 pgs: 97 active+clean; 659 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-16T11:06:13.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:13 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:06:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:13 smithi098 ceph-mon[35848]: pgmap v355: 97 pgs: 97 active+clean; 659 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-16T11:06:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:13 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:06:13.478 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:13 smithi153 ceph-mon[35521]: pgmap v355: 97 pgs: 97 active+clean; 659 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-16T11:06:13.478 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:13 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:06:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: pgmap v356: 97 pgs: 97 active+clean; 659 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-16T11:06:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:06:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:06:15.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: pgmap v356: 97 pgs: 97 active+clean; 659 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-16T11:06:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:06:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: pgmap v356: 97 pgs: 97 active+clean; 659 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-16T11:06:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:06:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:06:15.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:06:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:17 smithi098 ceph-mon[35848]: pgmap v357: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:06:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:17 smithi153 ceph-mon[35521]: pgmap v357: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:06:17.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:17 smithi078 ceph-mon[32235]: pgmap v357: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:06:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:19 smithi098 ceph-mon[35848]: pgmap v358: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-16T11:06:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:19 smithi153 ceph-mon[35521]: pgmap v358: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-16T11:06:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:19 smithi078 ceph-mon[32235]: pgmap v358: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-16T11:06:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:21 smithi098 ceph-mon[35848]: pgmap v359: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-16T11:06:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:21 smithi153 ceph-mon[35521]: pgmap v359: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-16T11:06:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:21 smithi078 ceph-mon[32235]: pgmap v359: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-16T11:06:23.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:23 smithi098 ceph-mon[35848]: pgmap v360: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-16T11:06:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:23 smithi153 ceph-mon[35521]: pgmap v360: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-16T11:06:23.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:23 smithi078 ceph-mon[32235]: pgmap v360: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-16T11:06:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:25 smithi098 ceph-mon[35848]: pgmap v361: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T11:06:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:25 smithi153 ceph-mon[35521]: pgmap v361: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T11:06:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:25 smithi078 ceph-mon[32235]: pgmap v361: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 -- 172.21.15.78:0/2778026781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 msgr2=0x7f965c100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2778026781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c100290 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9658009900 tx=0x7f965802bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 -- 172.21.15.78:0/2778026781 shutdown_connections 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2778026781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f965c101a10 0x7f965c108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2778026781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965c101050 0x7f965c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2778026781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c100290 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 -- 172.21.15.78:0/2778026781 >> 172.21.15.78:0/2778026781 conn(0x7f965c0fb430 msgr2=0x7f965c0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 -- 172.21.15.78:0/2778026781 shutdown_connections 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.391+0000 7f9663cd3640 1 -- 172.21.15.78:0/2778026781 wait complete. 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 Processor -- start 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 -- start start 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f965c101050 0x7f965c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965c101a10 0x7f965c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10b190 con 0x7f965c0ffe70 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10b010 con 0x7f965c101050 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9663cd3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10b310 con 0x7f965c101a10 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9661a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9661a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40574/0 (socket says 172.21.15.78:40574) 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9661a6f640 1 -- 172.21.15.78:0/2937505931 learned_addr learned my addr 172.21.15.78:0/2937505931 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f966126e640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f965c101050 0x7f965c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.392+0000 7f9662270640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965c101a10 0x7f965c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9661a6f640 1 -- 172.21.15.78:0/2937505931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965c101a10 msgr2=0x7f965c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9661a6f640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965c101a10 0x7f965c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9661a6f640 1 -- 172.21.15.78:0/2937505931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f965c101050 msgr2=0x7f965c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9661a6f640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f965c101050 0x7f965c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9661a6f640 1 -- 172.21.15.78:0/2937505931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c19f630 con 0x7f965c0ffe70 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f966126e640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f965c101050 0x7f965c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9661a6f640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c197a10 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9658000c00 tx=0x7f9658005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9652ffd640 1 -- 172.21.15.78:0/2937505931 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658043020 con 0x7f965c0ffe70 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9652ffd640 1 -- 172.21.15.78:0/2937505931 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9658004350 con 0x7f965c0ffe70 2024-03-16T11:06:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c19f8c0 con 0x7f965c0ffe70 2024-03-16T11:06:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9652ffd640 1 -- 172.21.15.78:0/2937505931 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658003b80 con 0x7f965c0ffe70 2024-03-16T11:06:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.393+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c19fe10 con 0x7f965c0ffe70 2024-03-16T11:06:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.395+0000 7f9652ffd640 1 -- 172.21.15.78:0/2937505931 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f9658046020 con 0x7f965c0ffe70 2024-03-16T11:06:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 Processor -- start 2024-03-16T11:06:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 -- start start 2024-03-16T11:06:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4340 0x55b23b2d4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2d9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b2dc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9652ffd640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96480a2940 0x7f96480a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b23b239a40 con 0x55b23b2d4d30 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b23b1e2920 con 0x55b23b2d9ea0 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.396+0000 7f9666995740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b23b235ae0 con 0x55b23b2d4340 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9661a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2d9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4340 0x55b23b2d4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4340 0x55b23b2d4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:46464/0 (socket says 172.21.15.78:46464) 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9661a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2d9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40588/0 (socket says 172.21.15.78:40588) 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f966126e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b2dc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 -- 172.21.15.78:0/1862124033 learned_addr learned my addr 172.21.15.78:0/1862124033 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96480a2940 0x7f96480a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9652ffd640 1 -- 172.21.15.78:0/2937505931 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f96580f0be0 con 0x7f965c0ffe70 2024-03-16T11:06:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 -- 172.21.15.78:0/1862124033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 msgr2=0x55b23b2dc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b2dc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 -- 172.21.15.78:0/1862124033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 msgr2=0x55b23b2d9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2d9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 -- 172.21.15.78:0/1862124033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b23b1fa460 con 0x55b23b2d4340 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f966126e640 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b2dc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9661a6f640 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2d9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4340 0x55b23b2d4760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f965400efb0 tx=0x7f965400c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcdd938f3a928906 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9662270640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96480a2940 0x7f96480a4e00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f965400cca0 tx=0x7f9654008000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9647fff640 1 -- 172.21.15.78:0/1862124033 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654021070 con 0x55b23b2d4340 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9647fff640 1 -- 172.21.15.78:0/1862124033 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9654003e20 con 0x55b23b2d4340 2024-03-16T11:06:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.397+0000 7f9647fff640 1 -- 172.21.15.78:0/1862124033 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654017040 con 0x55b23b2d4340 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 -- 172.21.15.78:0/1862124033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4340 msgr2=0x55b23b2d4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4340 0x55b23b2d4760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f965400efb0 tx=0x7f965400c7a0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 -- 172.21.15.78:0/1862124033 shutdown_connections 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b2dc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2d9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 --2- 172.21.15.78:0/1862124033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4340 0x55b23b2d4760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 -- 172.21.15.78:0/1862124033 >> 172.21.15.78:0/1862124033 conn(0x55b23b2cf8b0 msgr2=0x55b23b2cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 -- 172.21.15.78:0/1862124033 shutdown_connections 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.398+0000 7f9666995740 1 -- 172.21.15.78:0/1862124033 wait complete. 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 -- start start 2024-03-16T11:06:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 Processor -- start 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2de680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2debc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d9ea0 0x55b23b2e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b23b2dcdf0 con 0x55b23b2d4d30 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b23b1f5d70 con 0x55b23b2d4340 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b23b2dcf70 con 0x55b23b2d9ea0 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9666995740 10 client.0 _ll_register_callbacks cb 0x55b23af4da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f966126e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2debc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9662270640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d9ea0 0x55b23b2e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f966126e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2debc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40592/0 (socket says 172.21.15.78:40592) 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f966126e640 1 -- 172.21.15.78:0/834185201 learned_addr learned my addr 172.21.15.78:0/834185201 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.399+0000 7f9661a6f640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2de680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9662270640 1 -- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 msgr2=0x55b23b2de680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9662270640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2de680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9662270640 1 -- 172.21.15.78:0/834185201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 msgr2=0x55b23b2debc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9662270640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2debc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9662270640 1 -- 172.21.15.78:0/834185201 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b23af4e210 con 0x55b23b2d9ea0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f966126e640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2debc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9661a6f640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2de680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9662270640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d9ea0 0x55b23b2e5c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f965400a380 tx=0x7f965400e4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eacf3aa06aead6e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f96457fa640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f96457fa640 1 -- 172.21.15.78:0/834185201 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654032060 con 0x55b23b2d9ea0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f96457fa640 1 -- 172.21.15.78:0/834185201 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9654003b70 con 0x55b23b2d9ea0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9654003e20 con 0x55b23b2d9ea0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f96457fa640 1 -- 172.21.15.78:0/834185201 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9654021030 con 0x55b23b2d9ea0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9666995740 10 client.25303 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b23b01a620 con 0x55b23b2d9ea0 2024-03-16T11:06:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.400+0000 7f96457fa640 1 -- 172.21.15.78:0/834185201 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9654003e20 con 0x55b23b2d9ea0 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f96457fa640 1 -- 172.21.15.78:0/834185201 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f9654037070 con 0x55b23b2d9ea0 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9666995740 10 client.25303 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9666995740 10 client.25303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9666995740 20 client.25303 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9666995740 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b2ea260 0x55b23b2ec650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9666995740 4 client.25303 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b23b246f80 con 0x55b23b2ea260 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9661a6f640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b2ea260 0x55b23b2ec650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f9661a6f640 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b2ea260 0x55b23b2ec650 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edfc3cea5e196189 server_cookie=2602890dc8de30b0 in_seq=0 out_seq=0 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.401+0000 7f96457fa640 10 client.25303 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f96457fa640 1 -- 172.21.15.78:0/834185201 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b23b246f80 con 0x55b23b2ea260 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f96457fa640 10 client.25303 handle_command_reply: tid=0 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 client.25303 shutdown 2024-03-16T11:06:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d9ea0 msgr2=0x55b23b2e5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d9ea0 0x55b23b2e5c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f965400a380 tx=0x7f965400e4e0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 shutdown_connections 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b2ea260 0x55b23b2ec650 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d9ea0 0x55b23b2e5c00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4d30 0x55b23b2debc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 --2- 172.21.15.78:0/834185201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2de680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f96457fa640 0 client.25303 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 >> 172.21.15.78:0/834185201 conn(0x55b23b2cf8b0 msgr2=0x55b23b2cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 shutdown_connections 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 1 -- 172.21.15.78:0/834185201 wait complete. 2024-03-16T11:06:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.402+0000 7f9666995740 20 client.25303 trim_cache size 0 max 16384 2024-03-16T11:06:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 Processor -- start 2024-03-16T11:06:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 -- start start 2024-03-16T11:06:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2d60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b2d6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d9ea0 0x55b23b2ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1ab4b0 con 0x55b23b2d9ea0 2024-03-16T11:06:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1a8d80 con 0x55b23b2d4340 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9666995740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1a6bb0 con 0x55b23b2d4d30 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d9ea0 0x55b23b2ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d9ea0 0x55b23b2ce320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40604/0 (socket says 172.21.15.78:40604) 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 -- 172.21.15.78:0/3852385840 learned_addr learned my addr 172.21.15.78:0/3852385840 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9661a6f640 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2d60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f966126e640 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b2d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 -- 172.21.15.78:0/3852385840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 msgr2=0x55b23b2d6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b2d6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 -- 172.21.15.78:0/3852385840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 msgr2=0x55b23b2d60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2d60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.417+0000 7f9662270640 1 -- 172.21.15.78:0/3852385840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b23b2cb310 con 0x55b23b2d9ea0 2024-03-16T11:06:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9662270640 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d9ea0 0x55b23b2ce320 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f965400abf0 tx=0x7f965400e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e525158c36c29ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f96477fe640 1 -- 172.21.15.78:0/3852385840 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654032070 con 0x55b23b2d9ea0 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f96477fe640 1 -- 172.21.15.78:0/3852385840 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9654029070 con 0x55b23b2d9ea0 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f96477fe640 1 -- 172.21.15.78:0/3852385840 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965403d070 con 0x55b23b2d9ea0 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 -- 172.21.15.78:0/3852385840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d9ea0 msgr2=0x55b23b2ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d9ea0 0x55b23b2ce320 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f965400abf0 tx=0x7f965400e760 comp rx=0 tx=0).stop 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 -- 172.21.15.78:0/3852385840 shutdown_connections 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d9ea0 0x55b23b2ce320 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b2d6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 --2- 172.21.15.78:0/3852385840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d4340 0x55b23b2d60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 -- 172.21.15.78:0/3852385840 >> 172.21.15.78:0/3852385840 conn(0x55b23b2cf8b0 msgr2=0x55b23b2cfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.418+0000 7f9666995740 1 -- 172.21.15.78:0/3852385840 shutdown_connections 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 -- 172.21.15.78:0/3852385840 wait complete. 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 -- start start 2024-03-16T11:06:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 Processor -- start 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4340 0x55b23b34b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b34bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b352c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1a7240 con 0x55b23b2d4340 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1a86b0 con 0x55b23b2d9ea0 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1a7910 con 0x55b23b2d4d30 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9666995740 10 client.0 _ll_register_callbacks cb 0x7f964c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9661a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4340 0x55b23b34b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9661a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4340 0x55b23b34b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40618/0 (socket says 172.21.15.78:40618) 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9661a6f640 1 -- 172.21.15.78:0/4076532031 learned_addr learned my addr 172.21.15.78:0/4076532031 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f966126e640 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b34bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.419+0000 7f9662270640 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b352c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9661a6f640 1 -- 172.21.15.78:0/4076532031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 msgr2=0x55b23b34bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9661a6f640 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b34bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9661a6f640 1 -- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 msgr2=0x55b23b352c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9661a6f640 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b352c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9661a6f640 1 -- 172.21.15.78:0/4076532031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b23b353240 con 0x55b23b2d4340 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9661a6f640 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4340 0x55b23b34b6d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f96580096c0 tx=0x7f96580882f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ada07c35508e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f96457fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f96457fa640 1 -- 172.21.15.78:0/4076532031 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580bdc30 con 0x55b23b2d4340 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f96457fa640 1 -- 172.21.15.78:0/4076532031 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f96580ee7e0 con 0x55b23b2d4340 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9654029070 con 0x55b23b2d4340 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f96457fa640 1 -- 172.21.15.78:0/4076532031 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f96580bd530 con 0x55b23b2d4340 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9666995740 10 client.15489 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b23b354620 con 0x55b23b2d4340 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f96457fa640 1 -- 172.21.15.78:0/4076532031 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9654029070 con 0x55b23b2d4340 2024-03-16T11:06:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f96457fa640 1 -- 172.21.15.78:0/4076532031 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f96580bd790 con 0x55b23b2d4340 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9666995740 10 client.15489 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9666995740 10 client.15489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9666995740 20 client.15489 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.420+0000 7f9666995740 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b354fd0 0x55b23b3573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.421+0000 7f9666995740 4 client.15489 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.421+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b23b2412d0 con 0x55b23b354fd0 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.421+0000 7f966126e640 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b354fd0 0x55b23b3573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.421+0000 7f96457fa640 10 client.15489 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.421+0000 7f966126e640 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b354fd0 0x55b23b3573f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70bf642a8797f9a server_cookie=b251dd03efc5b342 in_seq=0 out_seq=0 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f96457fa640 1 -- 172.21.15.78:0/4076532031 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b23b2412d0 con 0x55b23b354fd0 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f96457fa640 10 client.15489 handle_command_reply: tid=0 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 client.15489 shutdown 2024-03-16T11:06:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4340 msgr2=0x55b23b34b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4340 0x55b23b34b6d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f96580096c0 tx=0x7f96580882f0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 shutdown_connections 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b23b354fd0 0x55b23b3573f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b23b2d9ea0 0x55b23b352c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23b2d4d30 0x55b23b34bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f96457fa640 0 client.15489 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 --2- 172.21.15.78:0/4076532031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b23b2d4340 0x55b23b34b6d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 >> 172.21.15.78:0/4076532031 conn(0x55b23b2cf8b0 msgr2=0x55b23b2de5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 shutdown_connections 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 1 -- 172.21.15.78:0/4076532031 wait complete. 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.424+0000 7f9666995740 20 client.15489 trim_cache size 0 max 16384 2024-03-16T11:06:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:06:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:06:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:06:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:06:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96480a2940 msgr2=0x7f96480a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96480a2940 0x7f96480a4e00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f965400cca0 tx=0x7f9654008000 comp rx=0 tx=0).stop 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 msgr2=0x7f965c197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c197a10 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9658000c00 tx=0x7f9658005dc0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 shutdown_connections 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96480a2940 0x7f96480a4e00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965c101a10 0x7f965c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f965c101050 0x7f965c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 --2- 172.21.15.78:0/2937505931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c0ffe70 0x7f965c197a10 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 >> 172.21.15.78:0/2937505931 conn(0x7f965c0fb430 msgr2=0x7f965c106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 shutdown_connections 2024-03-16T11:06:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:26.425+0000 7f9663cd3640 1 -- 172.21.15.78:0/2937505931 wait complete. 2024-03-16T11:06:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:27 smithi098 ceph-mon[35848]: pgmap v362: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-16T11:06:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:27 smithi153 ceph-mon[35521]: pgmap v362: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-16T11:06:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:27 smithi078 ceph-mon[32235]: pgmap v362: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-16T11:06:28.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:28 smithi153 ceph-mon[35521]: pgmap v363: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:06:28.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:28 smithi078 ceph-mon[32235]: pgmap v363: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:06:28.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:28 smithi098 ceph-mon[35848]: pgmap v363: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:06:30.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:06:30.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/2637397076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941018e0 msgr2=0x7f7c94101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/2637397076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941018e0 0x7f7c94101d60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009900 tx=0x7f7c9002bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/2637397076 shutdown_connections 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/2637397076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c941022a0 0x7f7c94108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/2637397076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941018e0 0x7f7c94101d60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/2637397076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c941006f0 0x7f7c94100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/2637397076 >> 172.21.15.78:0/2637397076 conn(0x7f7c940fbc80 msgr2=0x7f7c940fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/2637397076 shutdown_connections 2024-03-16T11:06:30.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.590+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/2637397076 wait complete. 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 Processor -- start 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 -- start start 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941006f0 0x7f7c94197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c941018e0 0x7f7c94197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c941022a0 0x7f7c9419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9410c200 con 0x7f7c941006f0 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9410c080 con 0x7f7c941018e0 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9c2b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9410c380 con 0x7f7c941022a0 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941006f0 0x7f7c94197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941006f0 0x7f7c94197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40626/0 (socket says 172.21.15.78:40626) 2024-03-16T11:06:30.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1693814896 learned_addr learned my addr 172.21.15.78:0/1693814896 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9984d640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c941018e0 0x7f7c94197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a84f640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c941022a0 0x7f7c9419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1693814896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c941022a0 msgr2=0x7f7c9419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c941022a0 0x7f7c9419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1693814896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c941018e0 msgr2=0x7f7c94197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c941018e0 0x7f7c94197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1693814896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9419f6a0 con 0x7f7c941006f0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941006f0 0x7f7c94197a80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800e9c0 tx=0x7f7c8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.591+0000 7f7c877fe640 1 -- 172.21.15.78:0/1693814896 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8800cd80 con 0x7f7c941006f0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.592+0000 7f7c877fe640 1 -- 172.21.15.78:0/1693814896 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c880035f0 con 0x7f7c941006f0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.592+0000 7f7c877fe640 1 -- 172.21.15.78:0/1693814896 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c88010690 con 0x7f7c941006f0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.592+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c9419f990 con 0x7f7c941006f0 2024-03-16T11:06:30.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.592+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c9419feb0 con 0x7f7c941006f0 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.594+0000 7f7c877fe640 1 -- 172.21.15.78:0/1693814896 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f7c880037a0 con 0x7f7c941006f0 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.594+0000 7f7c9ef7b740 1 Processor -- start 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.594+0000 7f7c9ef7b740 1 -- start start 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b8509a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b855a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b8560e0 0x55ae2b858570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ae2b7b1500 con 0x55ae2b8560e0 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ae2b759be0 con 0x55ae2b850f70 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ae2b7ad5a0 con 0x55ae2b850580 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c877fe640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7c800a2890 0x7f7c800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b8560e0 0x55ae2b858570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b8560e0 0x55ae2b858570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40638/0 (socket says 172.21.15.78:40638) 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9a04e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b855a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 -- 172.21.15.78:0/955998440 learned_addr learned my addr 172.21.15.78:0/955998440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9a84f640 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b8509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9a84f640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7c800a2890 0x7f7c800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c877fe640 1 -- 172.21.15.78:0/1693814896 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7c880cef80 con 0x7f7c941006f0 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 -- 172.21.15.78:0/955998440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 msgr2=0x55ae2b8509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b8509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 -- 172.21.15.78:0/955998440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 msgr2=0x55ae2b855a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b855a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 -- 172.21.15.78:0/955998440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae2b76d4b0 con 0x55ae2b8560e0 2024-03-16T11:06:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9984d640 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b8560e0 0x55ae2b858570 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009900 tx=0x7f7c90009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce862c6c5b8c1d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.595+0000 7f7c9a84f640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7c800a2890 0x7f7c800a4d50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c005c40 tx=0x7f7c8c009bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c84bfe640 1 -- 172.21.15.78:0/955998440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90045020 con 0x55ae2b8560e0 2024-03-16T11:06:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c84bfe640 1 -- 172.21.15.78:0/955998440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c90004040 con 0x55ae2b8560e0 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c84bfe640 1 -- 172.21.15.78:0/955998440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c900054e0 con 0x55ae2b8560e0 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/955998440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b8560e0 msgr2=0x55ae2b858570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b8560e0 0x55ae2b858570 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009900 tx=0x7f7c90009560 comp rx=0 tx=0).stop 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/955998440 shutdown_connections 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b8560e0 0x55ae2b858570 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b855a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/955998440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b8509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/955998440 >> 172.21.15.78:0/955998440 conn(0x55ae2b8476e0 msgr2=0x55ae2b847ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/955998440 shutdown_connections 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.596+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/955998440 wait complete. 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 -- start start 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 Processor -- start 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850580 0x55ae2b85a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b85af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b8560e0 0x55ae2b861f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ae2b859030 con 0x55ae2b850580 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ae2b76d160 con 0x55ae2b850f70 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ae2b8591b0 con 0x55ae2b8560e0 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9ef7b740 10 client.0 _ll_register_callbacks cb 0x55ae2b751150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850580 0x55ae2b85a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850580 0x55ae2b85a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40646/0 (socket says 172.21.15.78:40646) 2024-03-16T11:06:30.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1113564385 learned_addr learned my addr 172.21.15.78:0/1113564385 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:30.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a84f640 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b8560e0 0x55ae2b861f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9984d640 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b85af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1113564385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b8560e0 msgr2=0x55ae2b861f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b8560e0 0x55ae2b861f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 msgr2=0x55ae2b85af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b85af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.597+0000 7f7c9a04e640 1 -- 172.21.15.78:0/1113564385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae2b58f7d0 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850580 0x55ae2b85a9d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7c88098190 tx=0x7f7c880097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185636032691cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c7dffb640 1 -- 172.21.15.78:0/1113564385 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c88003b30 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c7dffb640 1 -- 172.21.15.78:0/1113564385 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7c880108f0 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c90004040 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c7dffb640 1 -- 172.21.15.78:0/1113564385 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7c8809bec0 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 10 client.15501 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ae2b4c53f0 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c7dffb640 1 -- 172.21.15.78:0/1113564385 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c90004040 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c7dffb640 1 -- 172.21.15.78:0/1113564385 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7c880a3ad0 con 0x55ae2b850580 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 10 client.15501 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:30.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 10 client.15501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 20 client.15501 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b866540 0x55ae2b8689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 4 client.15501 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae2b7bea40 con 0x55ae2b866540 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.598+0000 7f7c9984d640 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b866540 0x55ae2b8689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.599+0000 7f7c7dffb640 10 client.15501 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.599+0000 7f7c9984d640 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b866540 0x55ae2b8689d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ac70c4438d8b85 server_cookie=2f5b0363c30bdac8 in_seq=0 out_seq=0 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c7dffb640 1 -- 172.21.15.78:0/1113564385 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ae2b7bea40 con 0x55ae2b866540 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c7dffb640 10 client.15501 handle_command_reply: tid=0 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 client.15501 shutdown 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850580 msgr2=0x55ae2b85a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850580 0x55ae2b85a9d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7c88098190 tx=0x7f7c880097d0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 shutdown_connections 2024-03-16T11:06:30.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b866540 0x55ae2b8689d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b8560e0 0x55ae2b861f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b850f70 0x55ae2b85af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c7dffb640 0 client.15501 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/1113564385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850580 0x55ae2b85a9d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 >> 172.21.15.78:0/1113564385 conn(0x55ae2b8476e0 msgr2=0x55ae2b846900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 shutdown_connections 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/1113564385 wait complete. 2024-03-16T11:06:30.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.600+0000 7f7c9ef7b740 20 client.15501 trim_cache size 0 max 16384 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 Processor -- start 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 -- start start 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b861f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b846070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941a9340 con 0x55ae2b850f70 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941a6c10 con 0x55ae2b8560e0 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941a4a40 con 0x55ae2b850580 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9984d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b861f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9984d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b861f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40660/0 (socket says 172.21.15.78:40660) 2024-03-16T11:06:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9984d640 1 -- 172.21.15.78:0/2865525978 learned_addr learned my addr 172.21.15.78:0/2865525978 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.617+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9a84f640 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b846070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9984d640 1 -- 172.21.15.78:0/2865525978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 msgr2=0x55ae2b85acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9984d640 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9984d640 1 -- 172.21.15.78:0/2865525978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 msgr2=0x55ae2b846070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9984d640 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b846070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9984d640 1 -- 172.21.15.78:0/2865525978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae2b4850f0 con 0x55ae2b850f70 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9984d640 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b861f10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55ae2b848650 tx=0x7f7c9002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f56fe75b384638 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c7ffff640 1 -- 172.21.15.78:0/2865525978 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90045040 con 0x55ae2b850f70 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c7ffff640 1 -- 172.21.15.78:0/2865525978 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c900054f0 con 0x55ae2b850f70 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c7ffff640 1 -- 172.21.15.78:0/2865525978 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90004920 con 0x55ae2b850f70 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/2865525978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 msgr2=0x55ae2b861f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b861f10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55ae2b848650 tx=0x7f7c9002bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/2865525978 shutdown_connections 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b846070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b861f10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/2865525978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.618+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/2865525978 >> 172.21.15.78:0/2865525978 conn(0x55ae2b8476e0 msgr2=0x55ae2b85a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/2865525978 shutdown_connections 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/2865525978 wait complete. 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 -- start start 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 Processor -- start 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b85f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b860940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941a50d0 con 0x55ae2b850f70 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941a6540 con 0x55ae2b8560e0 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941a57a0 con 0x55ae2b850580 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9ef7b740 10 client.0 _ll_register_callbacks cb 0x7f7c880973f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b85f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b85f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40672/0 (socket says 172.21.15.78:40672) 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9a04e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 -- 172.21.15.78:0/3078552426 learned_addr learned my addr 172.21.15.78:0/3078552426 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9a84f640 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b860940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 -- 172.21.15.78:0/3078552426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 msgr2=0x55ae2b85f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 -- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 msgr2=0x55ae2b860940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b860940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.619+0000 7f7c9984d640 1 -- 172.21.15.78:0/3078552426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae2b7789d0 con 0x55ae2b850f70 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9984d640 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b85f930 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7c9002c170 tx=0x7f7c90005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91e2eecd4c0571e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c7dffb640 1 -- 172.21.15.78:0/3078552426 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90058030 con 0x55ae2b850f70 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c7dffb640 1 -- 172.21.15.78:0/3078552426 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7c90005040 con 0x55ae2b850f70 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c900054f0 con 0x55ae2b850f70 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c7dffb640 1 -- 172.21.15.78:0/3078552426 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7c90064050 con 0x55ae2b850f70 2024-03-16T11:06:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 10 client.15507 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ae2b8620a0 con 0x55ae2b850f70 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c7dffb640 1 -- 172.21.15.78:0/3078552426 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c900054f0 con 0x55ae2b850f70 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c7dffb640 1 -- 172.21.15.78:0/3078552426 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7c90041560 con 0x55ae2b850f70 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 10 client.15507 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 10 client.15507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 20 client.15507 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b8cc9d0 0x55ae2b8cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 4 client.15507 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ae2b7b8d90 con 0x55ae2b8cc9d0 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.620+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b8cc9d0 0x55ae2b8cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c7dffb640 10 client.15507 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9a04e640 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b8cc9d0 0x55ae2b8cedc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=296e12fa4d88180f server_cookie=bb68f334e0cd39bf in_seq=0 out_seq=0 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c7dffb640 1 -- 172.21.15.78:0/3078552426 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ae2b7b8d90 con 0x55ae2b8cc9d0 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c7dffb640 10 client.15507 handle_command_reply: tid=0 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 client.15507 shutdown 2024-03-16T11:06:30.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 msgr2=0x55ae2b85f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b85f930 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7c9002c170 tx=0x7f7c90005720 comp rx=0 tx=0).stop 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 shutdown_connections 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ae2b8cc9d0 0x55ae2b8cedc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ae2b8560e0 0x55ae2b860940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c7dffb640 0 client.15507 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ae2b850f70 0x55ae2b85f930 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 --2- 172.21.15.78:0/3078552426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae2b850580 0x55ae2b85f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 >> 172.21.15.78:0/3078552426 conn(0x55ae2b8476e0 msgr2=0x55ae2b84a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 shutdown_connections 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 1 -- 172.21.15.78:0/3078552426 wait complete. 2024-03-16T11:06:30.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.621+0000 7f7c9ef7b740 20 client.15507 trim_cache size 0 max 16384 2024-03-16T11:06:30.631 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:06:30.631 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-16T11:06:30.631 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-16T11:06:30.631 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7c800a2890 msgr2=0x7f7c800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7c800a2890 0x7f7c800a4d50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c005c40 tx=0x7f7c8c009bb0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941006f0 msgr2=0x7f7c94197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941006f0 0x7f7c94197a80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800e9c0 tx=0x7f7c8800ee80 comp rx=0 tx=0).stop 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 shutdown_connections 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7c800a2890 0x7f7c800a4d50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c941022a0 0x7f7c9419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c941018e0 0x7f7c94197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 --2- 172.21.15.78:0/1693814896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c941006f0 0x7f7c94197a80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 >> 172.21.15.78:0/1693814896 conn(0x7f7c940fbc80 msgr2=0x7f7c94069e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 shutdown_connections 2024-03-16T11:06:30.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.622+0000 7f7c9c2b2640 1 -- 172.21.15.78:0/1693814896 wait complete. 2024-03-16T11:06:30.634 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T11:06:30.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:06:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.734+0000 7f1c92b52640 1 -- 172.21.15.78:0/1180897483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c102b60 msgr2=0x7f1c8c102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.734+0000 7f1c92b52640 1 --2- 172.21.15.78:0/1180897483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c102b60 0x7f1c8c102fe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009890 tx=0x7f1c7802bb70 comp rx=0 tx=0).stop 2024-03-16T11:06:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- 172.21.15.78:0/1180897483 shutdown_connections 2024-03-16T11:06:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 --2- 172.21.15.78:0/1180897483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c103520 0x7f1c8c109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 --2- 172.21.15.78:0/1180897483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c102b60 0x7f1c8c102fe0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 --2- 172.21.15.78:0/1180897483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c0ffaf0 0x7f1c8c0fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- 172.21.15.78:0/1180897483 >> 172.21.15.78:0/1180897483 conn(0x7f1c8c0fb400 msgr2=0x7f1c8c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- 172.21.15.78:0/1180897483 shutdown_connections 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- 172.21.15.78:0/1180897483 wait complete. 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 Processor -- start 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- start start 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c0ffaf0 0x7f1c8c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c102b60 0x7f1c8c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103520 0x7f1c8c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c10d330 con 0x7f1c8c103520 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c10d1b0 con 0x7f1c8c0ffaf0 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c92b52640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c10d4b0 con 0x7f1c8c102b60 2024-03-16T11:06:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c910ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103520 0x7f1c8c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c910ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103520 0x7f1c8c19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40682/0 (socket says 172.21.15.78:40682) 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c908ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c0ffaf0 0x7f1c8c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c910ef640 1 -- 172.21.15.78:0/485659319 learned_addr learned my addr 172.21.15.78:0/485659319 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.735+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c102b60 0x7f1c8c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c910ef640 1 -- 172.21.15.78:0/485659319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c102b60 msgr2=0x7f1c8c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c910ef640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c102b60 0x7f1c8c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c910ef640 1 -- 172.21.15.78:0/485659319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c0ffaf0 msgr2=0x7f1c8c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c910ef640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c0ffaf0 0x7f1c8c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c910ef640 1 -- 172.21.15.78:0/485659319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c19f680 con 0x7f1c8c103520 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c908ee640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c0ffaf0 0x7f1c8c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c910ef640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103520 0x7f1c8c19f010 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400efd0 tx=0x7f1c8400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c89ffb640 1 -- 172.21.15.78:0/485659319 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84003b70 con 0x7f1c8c103520 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8c19f970 con 0x7f1c8c103520 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c89ffb640 1 -- 172.21.15.78:0/485659319 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c84003d10 con 0x7f1c8c103520 2024-03-16T11:06:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c89ffb640 1 -- 172.21.15.78:0/485659319 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84010850 con 0x7f1c8c103520 2024-03-16T11:06:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.736+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8c19fe90 con 0x7f1c8c103520 2024-03-16T11:06:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.737+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8c108d00 con 0x7f1c8c103520 2024-03-16T11:06:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.738+0000 7f1c89ffb640 1 -- 172.21.15.78:0/485659319 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f1c84004000 con 0x7f1c8c103520 2024-03-16T11:06:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.739+0000 7f1c89ffb640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c7c0a2890 0x7f1c7c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.739+0000 7f1c908ee640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c7c0a2890 0x7f1c7c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.739+0000 7f1c89ffb640 1 -- 172.21.15.78:0/485659319 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1c840cf000 con 0x7f1c8c103520 2024-03-16T11:06:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.739+0000 7f1c908ee640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c7c0a2890 0x7f1c7c0a4d50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c069bd0 tx=0x7f1c80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.743+0000 7f1c89ffb640 1 -- 172.21.15.78:0/485659319 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c8409bf40 con 0x7f1c8c103520 2024-03-16T11:06:30.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.956+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c8c02d1b0 con 0x7f1c8c103520 2024-03-16T11:06:30.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.956+0000 7f1c89ffb640 1 -- 172.21.15.78:0/485659319 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7f1c8409b840 con 0x7f1c8c103520 2024-03-16T11:06:30.957 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:06:30.957 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:06:30.958 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:06:30.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c7c0a2890 msgr2=0x7f1c7c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c7c0a2890 0x7f1c7c0a4d50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c069bd0 tx=0x7f1c80009260 comp rx=0 tx=0).stop 2024-03-16T11:06:30.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103520 msgr2=0x7f1c8c19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:30.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103520 0x7f1c8c19f010 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400efd0 tx=0x7f1c8400c6a0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 shutdown_connections 2024-03-16T11:06:30.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103520 0x7f1c8c19f010 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1c7c0a2890 0x7f1c7c0a4d50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c102b60 0x7f1c8c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 --2- 172.21.15.78:0/485659319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c8c0ffaf0 0x7f1c8c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:30.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 >> 172.21.15.78:0/485659319 conn(0x7f1c8c0fb400 msgr2=0x7f1c8c105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:30.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 shutdown_connections 2024-03-16T11:06:30.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:30.958+0000 7f1c92b52640 1 -- 172.21.15.78:0/485659319 wait complete. 2024-03-16T11:06:30.970 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T11:06:31.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/3545096323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 msgr2=0x7fc5a4101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/3545096323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a4101d50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc598009900 tx=0x7fc59802bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/3545096323 shutdown_connections 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/3545096323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a4102290 0x7fc5a4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/3545096323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a4101d50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/3545096323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a41006e0 0x7fc5a4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/3545096323 >> 172.21.15.78:0/3545096323 conn(0x7fc5a40fbc70 msgr2=0x7fc5a40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.074+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/3545096323 shutdown_connections 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/3545096323 wait complete. 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 Processor -- start 2024-03-16T11:06:31.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 -- start start 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a41006e0 0x7fc5a4199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a4102290 0x7fc5a41a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a410c210 con 0x7fc5a41018d0 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a410c090 con 0x7fc5a41006e0 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a9eb5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a410c390 con 0x7fc5a4102290 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.075+0000 7fc5a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40698/0 (socket says 172.21.15.78:40698) 2024-03-16T11:06:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1352074250 learned_addr learned my addr 172.21.15.78:0/1352074250 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a41006e0 0x7fc5a4199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a3fff640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a4102290 0x7fc5a41a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1352074250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a4102290 msgr2=0x7fc5a41a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a4102290 0x7fc5a41a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1352074250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a41006e0 msgr2=0x7fc5a4199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a41006e0 0x7fc5a4199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1352074250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a41a1880 con 0x7fc5a41018d0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a419a1b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc598006010 tx=0x7fc598005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a0ff9640 1 -- 172.21.15.78:0/1352074250 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598043020 con 0x7fc5a41018d0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a0ff9640 1 -- 172.21.15.78:0/1352074250 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5980043d0 con 0x7fc5a41018d0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a41a1b10 con 0x7fc5a41018d0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a0ff9640 1 -- 172.21.15.78:0/1352074250 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598003b80 con 0x7fc5a41018d0 2024-03-16T11:06:31.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.076+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a41a2060 con 0x7fc5a41018d0 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.078+0000 7fc5a0ff9640 1 -- 172.21.15.78:0/1352074250 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fc598046020 con 0x7fc5a41018d0 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 Processor -- start 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 -- start start 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c3175ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175f240 0x557c31763d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c31766840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557c316c3490 con 0x557c3175e850 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557c3166bc90 con 0x557c317643b0 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5acb77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557c316bf530 con 0x557c3175f240 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c3175ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a0ff9640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc58c0a2890 0x7fc58c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c3175ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40706/0 (socket says 172.21.15.78:40706) 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a3fff640 1 -- 172.21.15.78:0/456281388 learned_addr learned my addr 172.21.15.78:0/456281388 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175f240 0x557c31763d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c31766840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a3fff640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc58c0a2890 0x7fc58c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a37fe640 1 -- 172.21.15.78:0/456281388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 msgr2=0x557c31766840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c31766840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a37fe640 1 -- 172.21.15.78:0/456281388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 msgr2=0x557c3175ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c3175ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a37fe640 1 -- 172.21.15.78:0/456281388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c31726500 con 0x557c3175f240 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a3fff640 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c3175ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.079+0000 7fc5a0ff9640 1 -- 172.21.15.78:0/1352074250 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc5980f0d80 con 0x7fc5a41018d0 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175f240 0x557c31763d40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4101730 tx=0x7fc594011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d784aee131e1453 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5a3fff640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc58c0a2890 0x7fc58c0a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x557c3175faa0 tx=0x7fc59c004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc589ffb640 1 -- 172.21.15.78:0/456281388 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594009d70 con 0x557c3175f240 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc589ffb640 1 -- 172.21.15.78:0/456281388 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5940035f0 con 0x557c3175f240 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc589ffb640 1 -- 172.21.15.78:0/456281388 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594019630 con 0x557c3175f240 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 -- 172.21.15.78:0/456281388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175f240 msgr2=0x557c31763d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175f240 0x557c31763d40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4101730 tx=0x7fc594011810 comp rx=0 tx=0).stop 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 -- 172.21.15.78:0/456281388 shutdown_connections 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c31766840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175f240 0x557c31763d40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 --2- 172.21.15.78:0/456281388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c3175ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 -- 172.21.15.78:0/456281388 >> 172.21.15.78:0/456281388 conn(0x557c31759dc0 msgr2=0x557c3175a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 -- 172.21.15.78:0/456281388 shutdown_connections 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.080+0000 7fc5acb77740 1 -- 172.21.15.78:0/456281388 wait complete. 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 -- start start 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 Processor -- start 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c31768d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c317692a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c317702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557c31767360 con 0x557c3175f240 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557c3167f230 con 0x557c317643b0 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557c317674e0 con 0x557c3175e850 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5acb77740 10 client.0 _ll_register_callbacks cb 0x557c316bb010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c317692a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c317692a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40722/0 (socket says 172.21.15.78:40722) 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/2943656072 learned_addr learned my addr 172.21.15.78:0/2943656072 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c31768d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.081+0000 7fc5a3fff640 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c317702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/2943656072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 msgr2=0x557c31768d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c31768d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 msgr2=0x557c317702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c317702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/2943656072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c3168e100 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c317692a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc598000c00 tx=0x7fc5980b92a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7846f0d311f805f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc582ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc582ffd640 1 -- 172.21.15.78:0/2943656072 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5980a8070 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc582ffd640 1 -- 172.21.15.78:0/2943656072 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc5980054b0 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5940035f0 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc582ffd640 1 -- 172.21.15.78:0/2943656072 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc5980056b0 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 10 client.15525 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557c313d73f0 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc582ffd640 1 -- 172.21.15.78:0/2943656072 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5940035f0 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc582ffd640 1 -- 172.21.15.78:0/2943656072 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc5980a6bd0 con 0x557c3175f240 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 10 client.15525 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 10 client.15525 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 20 client.15525 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c31774b90 0x557c31777020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 4 client.15525 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.082+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c316d09d0 con 0x557c31774b90 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.083+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c31774b90 0x557c31777020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.083+0000 7fc582ffd640 10 client.15525 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.083+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c31774b90 0x557c31777020 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=61d7f51bcadb990e server_cookie=79038330472781e5 in_seq=0 out_seq=0 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc582ffd640 1 -- 172.21.15.78:0/2943656072 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c316d09d0 con 0x557c31774b90 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc582ffd640 10 client.15525 handle_command_reply: tid=0 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 client.15525 shutdown 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 msgr2=0x557c317692a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c317692a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc598000c00 tx=0x7fc5980b92a0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 shutdown_connections 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c31774b90 0x557c31777020 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c317702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c317692a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc582ffd640 0 client.15525 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:06:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2943656072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c31768d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 >> 172.21.15.78:0/2943656072 conn(0x557c31759dc0 msgr2=0x557c31758fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 shutdown_connections 2024-03-16T11:06:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 1 -- 172.21.15.78:0/2943656072 wait complete. 2024-03-16T11:06:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.084+0000 7fc5acb77740 20 client.15525 trim_cache size 0 max 16384 2024-03-16T11:06:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 Processor -- start 2024-03-16T11:06:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 -- start start 2024-03-16T11:06:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c31758750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c3175f240 0x557c31770260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c317643b0 0x557c31758c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a41ad5d0 con 0x557c3175e850 2024-03-16T11:06:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a41aaea0 con 0x557c3175f240 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5acb77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a41a8cd0 con 0x557c317643b0 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c31758750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c31758750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40732/0 (socket says 172.21.15.78:40732) 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.097+0000 7fc5a37fe640 1 -- 172.21.15.78:0/2584958897 learned_addr learned my addr 172.21.15.78:0/2584958897 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c3175f240 0x557c31770260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a3fff640 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c317643b0 0x557c31758c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a37fe640 1 -- 172.21.15.78:0/2584958897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c317643b0 msgr2=0x557c31758c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c317643b0 0x557c31758c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a37fe640 1 -- 172.21.15.78:0/2584958897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c3175f240 msgr2=0x557c31770260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c3175f240 0x557c31770260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a37fe640 1 -- 172.21.15.78:0/2584958897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c314a17d0 con 0x557c3175e850 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c3175f240 0x557c31770260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:06:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c31758750 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc594011d10 tx=0x7fc594005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3938663666d8f3f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5897fa640 1 -- 172.21.15.78:0/2584958897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940097a0 con 0x557c3175e850 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5897fa640 1 -- 172.21.15.78:0/2584958897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc594003cd0 con 0x557c3175e850 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5897fa640 1 -- 172.21.15.78:0/2584958897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940055e0 con 0x557c3175e850 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5acb77740 1 -- 172.21.15.78:0/2584958897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 msgr2=0x557c31758750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c31758750 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc594011d10 tx=0x7fc594005d70 comp rx=0 tx=0).stop 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5acb77740 1 -- 172.21.15.78:0/2584958897 shutdown_connections 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.098+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c317643b0 0x557c31758c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c3175f240 0x557c31770260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 --2- 172.21.15.78:0/2584958897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175e850 0x557c31758750 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 -- 172.21.15.78:0/2584958897 >> 172.21.15.78:0/2584958897 conn(0x557c31759dc0 msgr2=0x557c3175a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 -- 172.21.15.78:0/2584958897 shutdown_connections 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 -- 172.21.15.78:0/2584958897 wait complete. 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 -- start start 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 Processor -- start 2024-03-16T11:06:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c3176d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c3176dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c3176ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a41a9360 con 0x557c3175f240 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a41aa7d0 con 0x557c317643b0 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a41a9a30 con 0x557c3175e850 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5acb77740 10 client.0 _ll_register_callbacks cb 0x7fc598004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c3176dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c3176ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c3176dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40748/0 (socket says 172.21.15.78:40748) 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c3176ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:46462/0 (socket says 172.21.15.78:46462) 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1969122603 learned_addr learned my addr 172.21.15.78:0/1969122603 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.099+0000 7fc5a37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c3176d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1969122603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 msgr2=0x557c3176d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c3176d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1969122603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 msgr2=0x557c3176ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c3176ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5a2ffd640 1 -- 172.21.15.78:0/1969122603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c3176f360 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5a2ffd640 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c3176dd60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc5980036a0 tx=0x7fc5980b95f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92ce5a78081e40d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc582ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc582ffd640 1 -- 172.21.15.78:0/1969122603 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5980bb810 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc582ffd640 1 -- 172.21.15.78:0/1969122603 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc5980054b0 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc594003cd0 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc582ffd640 1 -- 172.21.15.78:0/1969122603 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc5980bddd0 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 10 client.15531 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557c3176f360 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc582ffd640 1 -- 172.21.15.78:0/1969122603 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc594003cd0 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc582ffd640 1 -- 172.21.15.78:0/1969122603 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc5980c2740 con 0x557c3175f240 2024-03-16T11:06:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 10 client.15531 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 10 client.15531 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 20 client.15531 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c317db440 0x557c317dd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 4 client.15531 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c316cad20 con 0x557c317db440 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.100+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c317db440 0x557c317dd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc582ffd640 10 client.15531 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5a37fe640 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c317db440 0x557c317dd860 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=381bcad670d5641 server_cookie=b505c30bc67c4c58 in_seq=0 out_seq=0 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc582ffd640 1 -- 172.21.15.78:0/1969122603 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c316cad20 con 0x557c317db440 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc582ffd640 10 client.15531 handle_command_reply: tid=0 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 client.15531 shutdown 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 msgr2=0x557c3176dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c3176dd60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc5980036a0 tx=0x7fc5980b95f0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 shutdown_connections 2024-03-16T11:06:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x557c317db440 0x557c317dd860 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557c317643b0 0x557c3176ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc582ffd640 0 client.15531 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c3175f240 0x557c3176dd60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 --2- 172.21.15.78:0/1969122603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557c3175e850 0x557c3176d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 >> 172.21.15.78:0/1969122603 conn(0x557c31759dc0 msgr2=0x557c3175d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 shutdown_connections 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 1 -- 172.21.15.78:0/1969122603 wait complete. 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.101+0000 7fc5acb77740 20 client.15531 trim_cache size 0 max 16384 2024-03-16T11:06:31.109 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:06:31.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc58c0a2890 msgr2=0x7fc58c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc58c0a2890 0x7fc58c0a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x557c3175faa0 tx=0x7fc59c004ab0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 msgr2=0x7fc5a419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a419a1b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc598006010 tx=0x7fc598005e40 comp rx=0 tx=0).stop 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 shutdown_connections 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc58c0a2890 0x7fc58c0a4d50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a4102290 0x7fc5a41a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a41018d0 0x7fc5a419a1b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 --2- 172.21.15.78:0/1352074250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a41006e0 0x7fc5a4199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 >> 172.21.15.78:0/1352074250 conn(0x7fc5a40fbc70 msgr2=0x7fc5a4068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 shutdown_connections 2024-03-16T11:06:31.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.102+0000 7fc5a9eb5640 1 -- 172.21.15.78:0/1352074250 wait complete. 2024-03-16T11:06:31.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 -- 172.21.15.78:0/4065948561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc100900 msgr2=0x7fd7dc100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 --2- 172.21.15.78:0/4065948561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc100900 0x7fd7dc100d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009860 tx=0x7fd7d802bb70 comp rx=0 tx=0).stop 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 -- 172.21.15.78:0/4065948561 shutdown_connections 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 --2- 172.21.15.78:0/4065948561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7dc1012a0 0x7fd7dc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 --2- 172.21.15.78:0/4065948561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc100900 0x7fd7dc100d60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 --2- 172.21.15.78:0/4065948561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101cb0 0x7fd7dc1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 -- 172.21.15.78:0/4065948561 >> 172.21.15.78:0/4065948561 conn(0x7fd7dc0fb410 msgr2=0x7fd7dc0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 -- 172.21.15.78:0/4065948561 shutdown_connections 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 -- 172.21.15.78:0/4065948561 wait complete. 2024-03-16T11:06:31.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.216+0000 7fd7e455b640 1 Processor -- start 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 -- start start 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7dc100900 0x7fd7dc100380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc1012a0 0x7fd7dc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc101cb0 0x7fd7dc19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc10c1f0 con 0x7fd7dc1012a0 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc10c070 con 0x7fd7dc101cb0 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc10c370 con 0x7fd7dc100900 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc1012a0 0x7fd7dc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc1012a0 0x7fd7dc19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40768/0 (socket says 172.21.15.78:40768) 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2948820522 learned_addr learned my addr 172.21.15.78:0/2948820522 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e2af8640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc101cb0 0x7fd7dc19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7dc100900 0x7fd7dc100380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2948820522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7dc100900 msgr2=0x7fd7dc100380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7dc100900 0x7fd7dc100380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2948820522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc101cb0 msgr2=0x7fd7dc19ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc101cb0 0x7fd7dc19ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2948820522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc19f620 con 0x7fd7dc1012a0 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7dc100900 0x7fd7dc100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc1012a0 0x7fd7dc19c720 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005940 tx=0x7fd7d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7d37fe640 1 -- 172.21.15.78:0/2948820522 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8041020 con 0x7fd7dc1012a0 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7dc19f8b0 con 0x7fd7dc1012a0 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7d37fe640 1 -- 172.21.15.78:0/2948820522 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7d802c9c0 con 0x7fd7dc1012a0 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7d37fe640 1 -- 172.21.15.78:0/2948820522 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8039040 con 0x7fd7dc1012a0 2024-03-16T11:06:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.217+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7dc19fdf0 con 0x7fd7dc1012a0 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.220+0000 7fd7d37fe640 1 -- 172.21.15.78:0/2948820522 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fd7d8044020 con 0x7fd7dc1012a0 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.220+0000 7fd7e7228740 1 Processor -- start 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.220+0000 7fd7e7228740 1 -- start start 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.220+0000 7fd7d37fe640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd7c80a2890 0x7fd7c80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.220+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f152b850 0x5637f152bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.220+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1530d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.220+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f15313b0 0x5637f1533840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e7228740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5637f1490490 con 0x5637f152c240 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e7228740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637f1438c90 con 0x5637f15313b0 2024-03-16T11:06:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e7228740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5637f148c530 con 0x5637f152b850 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1530d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e2af8640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd7c80a2890 0x7fd7c80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1530d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40770/0 (socket says 172.21.15.78:40770) 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2893527700 learned_addr learned my addr 172.21.15.78:0/2893527700 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f152b850 0x5637f152bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7d37fe640 1 -- 172.21.15.78:0/2948820522 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd7d80f04c0 con 0x7fd7dc1012a0 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e2af8640 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f15313b0 0x5637f1533840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2893527700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f152b850 msgr2=0x5637f152bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f152b850 0x5637f152bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2893527700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f15313b0 msgr2=0x5637f1533840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f15313b0 0x5637f1533840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 -- 172.21.15.78:0/2893527700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f14f3500 con 0x5637f152c240 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f152b850 0x5637f152bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e2af8640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd7c80a2890 0x7fd7c80a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4005fd0 tx=0x7fd7d4009e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1530d40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80b9270 tx=0x7fd7d802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d552a9cff6cdf16e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7d0bfe640 1 -- 172.21.15.78:0/2893527700 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80bc770 con 0x5637f152c240 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7d0bfe640 1 -- 172.21.15.78:0/2893527700 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7d80bd510 con 0x5637f152c240 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.221+0000 7fd7d0bfe640 1 -- 172.21.15.78:0/2893527700 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d803a250 con 0x5637f152c240 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 -- 172.21.15.78:0/2893527700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 msgr2=0x5637f1530d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1530d40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80b9270 tx=0x7fd7d802bb70 comp rx=0 tx=0).stop 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 -- 172.21.15.78:0/2893527700 shutdown_connections 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f15313b0 0x5637f1533840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1530d40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 --2- 172.21.15.78:0/2893527700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f152b850 0x5637f152bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 -- 172.21.15.78:0/2893527700 >> 172.21.15.78:0/2893527700 conn(0x5637f1526dc0 msgr2=0x5637f15271c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 -- 172.21.15.78:0/2893527700 shutdown_connections 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.222+0000 7fd7e7228740 1 -- 172.21.15.78:0/2893527700 wait complete. 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 -- start start 2024-03-16T11:06:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 Processor -- start 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f1535a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1535f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f153cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5637f1490490 con 0x5637f152c240 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5637f144c230 con 0x5637f152b850 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5637f1534120 con 0x5637f15313b0 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e7228740 10 client.0 _ll_register_callbacks cb 0x5637f1488010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1535f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e22f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f1535a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1535f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40786/0 (socket says 172.21.15.78:40786) 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e22f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f1535a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:46522/0 (socket says 172.21.15.78:46522) 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 -- 172.21.15.78:0/3280373188 learned_addr learned my addr 172.21.15.78:0/3280373188 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e2af8640 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f153cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 -- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 msgr2=0x5637f153cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f153cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 -- 172.21.15.78:0/3280373188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 msgr2=0x5637f1535a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f1535a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 -- 172.21.15.78:0/3280373188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637f145b100 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f1535a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1535f60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80b97e0 tx=0x7fd7d8005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e79206d6ace1ca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.223+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3280373188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d808ed40 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3280373188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd7d8003760 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7d80bd510 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3280373188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd7d808f7b0 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 10 client.15543 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5637f11a43f0 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3280373188 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7d80bd510 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3280373188 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd7d80d4020 con 0x5637f152c240 2024-03-16T11:06:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 10 client.15543 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 10 client.15543 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 20 client.15543 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15415a0 0x5637f1543990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 4 client.15543 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637f149d9d0 con 0x5637f15415a0 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.224+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15415a0 0x5637f1543990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15415a0 0x5637f1543990 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a4f9e98d92e8d0e server_cookie=8dc05f506a641bb7 in_seq=0 out_seq=0 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7c5ffb640 10 client.15543 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3280373188 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5637f149d9d0 con 0x5637f15415a0 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7c5ffb640 10 client.15543 handle_command_reply: tid=0 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 client.15543 shutdown 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 msgr2=0x5637f1535f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1535f60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80b97e0 tx=0x7fd7d8005750 comp rx=0 tx=0).stop 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 shutdown_connections 2024-03-16T11:06:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15415a0 0x5637f1543990 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f153cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7c5ffb640 0 client.15543 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f1535f60 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3280373188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f1535a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 >> 172.21.15.78:0/3280373188 conn(0x5637f1526dc0 msgr2=0x5637f1525fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 shutdown_connections 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.225+0000 7fd7e7228740 1 -- 172.21.15.78:0/3280373188 wait complete. 2024-03-16T11:06:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.226+0000 7fd7e7228740 20 client.15543 trim_cache size 0 max 16384 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 Processor -- start 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 -- start start 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f152ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f153cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f152d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1ab500 con 0x5637f152c240 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1a8dd0 con 0x5637f152b850 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e7228740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1a6c00 con 0x5637f15313b0 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f153cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f153cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40798/0 (socket says 172.21.15.78:40798) 2024-03-16T11:06:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 -- 172.21.15.78:0/798589772 learned_addr learned my addr 172.21.15.78:0/798589772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f152ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e2af8640 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f152d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 -- 172.21.15.78:0/798589772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 msgr2=0x5637f152d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f152d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 -- 172.21.15.78:0/798589772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 msgr2=0x5637f152ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f152ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 -- 172.21.15.78:0/798589772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f126e7d0 con 0x5637f152c240 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f152ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f153cef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80bcb10 tx=0x7fd7d80bca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d039fbe5dcace890 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7c7fff640 1 -- 172.21.15.78:0/798589772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8085160 con 0x5637f152c240 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7c7fff640 1 -- 172.21.15.78:0/798589772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7d80bd510 con 0x5637f152c240 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.239+0000 7fd7c7fff640 1 -- 172.21.15.78:0/798589772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8085380 con 0x5637f152c240 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 -- 172.21.15.78:0/798589772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 msgr2=0x5637f153cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f153cef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80bcb10 tx=0x7fd7d80bca30 comp rx=0 tx=0).stop 2024-03-16T11:06:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 -- 172.21.15.78:0/798589772 shutdown_connections 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f152d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152c240 0x5637f153cef0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 --2- 172.21.15.78:0/798589772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152b850 0x5637f152ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 -- 172.21.15.78:0/798589772 >> 172.21.15.78:0/798589772 conn(0x5637f1526dc0 msgr2=0x5637f15344c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 -- 172.21.15.78:0/798589772 shutdown_connections 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 -- 172.21.15.78:0/798589772 wait complete. 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 -- start start 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.240+0000 7fd7e7228740 1 Processor -- start 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152b850 0x5637f153cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152c240 0x5637f1539740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f1539c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1a7290 con 0x5637f152b850 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1a8700 con 0x5637f152c240 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1a7960 con 0x5637f15313b0 2024-03-16T11:06:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 10 client.0 _ll_register_callbacks cb 0x7fd7d40137c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152b850 0x5637f153cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152b850 0x5637f153cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40810/0 (socket says 172.21.15.78:40810) 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 -- 172.21.15.78:0/3129851472 learned_addr learned my addr 172.21.15.78:0/3129851472 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152c240 0x5637f1539740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e2af8640 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f1539c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 -- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 msgr2=0x5637f1539c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f1539c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 -- 172.21.15.78:0/3129851472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152c240 msgr2=0x5637f1539740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152c240 0x5637f1539740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 -- 172.21.15.78:0/3129851472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637f153a2c0 con 0x5637f152b850 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e22f7640 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152b850 0x5637f153cca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x5637f152caa0 tx=0x7fd7cc012cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b556927fb802ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3129851472 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc004440 con 0x5637f152b850 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3129851472 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd7cc01f930 con 0x5637f152b850 2024-03-16T11:06:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7d80bd510 con 0x5637f152b850 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3129851472 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd7cc01fb30 con 0x5637f152b850 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 10 client.15549 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.241+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5637f15a69a0 con 0x5637f152b850 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3129851472 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7d80bd510 con 0x5637f152b850 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3129851472 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd7cc0035f0 con 0x5637f152b850 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e7228740 10 client.15549 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e7228740 10 client.15549 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e7228740 20 client.15549 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15a73a0 0x5637f15a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e7228740 4 client.15549 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5637f1497d20 con 0x5637f15a73a0 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15a73a0 0x5637f15a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7c5ffb640 10 client.15549 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e1af6640 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15a73a0 0x5637f15a97c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=16ecc352c8644829 server_cookie=3844ee2589cac45b in_seq=0 out_seq=0 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7c5ffb640 1 -- 172.21.15.78:0/3129851472 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5637f1497d20 con 0x5637f15a73a0 2024-03-16T11:06:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7c5ffb640 10 client.15549 handle_command_reply: tid=0 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.242+0000 7fd7e7228740 1 client.15549 shutdown 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152b850 msgr2=0x5637f153cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152b850 0x5637f153cca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x5637f152caa0 tx=0x7fd7cc012cc0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 shutdown_connections 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5637f15a73a0 0x5637f15a97c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5637f15313b0 0x5637f1539c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5637f152c240 0x5637f1539740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7c5ffb640 0 client.15549 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 --2- 172.21.15.78:0/3129851472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5637f152b850 0x5637f153cca0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 >> 172.21.15.78:0/3129851472 conn(0x5637f1526dc0 msgr2=0x5637f152ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 shutdown_connections 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 1 -- 172.21.15.78:0/3129851472 wait complete. 2024-03-16T11:06:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e7228740 20 client.15549 trim_cache size 0 max 16384 2024-03-16T11:06:31.251 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd7c80a2890 msgr2=0x7fd7c80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd7c80a2890 0x7fd7c80a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4005fd0 tx=0x7fd7d4009e00 comp rx=0 tx=0).stop 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc1012a0 msgr2=0x7fd7dc19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc1012a0 0x7fd7dc19c720 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005940 tx=0x7fd7d8004270 comp rx=0 tx=0).stop 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 shutdown_connections 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7dc101cb0 0x7fd7dc19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc1012a0 0x7fd7dc19c720 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fd7c80a2890 0x7fd7c80a4d50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 --2- 172.21.15.78:0/2948820522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7dc100900 0x7fd7dc100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 >> 172.21.15.78:0/2948820522 conn(0x7fd7dc0fb410 msgr2=0x7fd7dc104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 shutdown_connections 2024-03-16T11:06:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.243+0000 7fd7e455b640 1 -- 172.21.15.78:0/2948820522 wait complete. 2024-03-16T11:06:31.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:31 smithi078 ceph-mon[32235]: pgmap v364: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-16T11:06:31.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:31 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/485659319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:06:31.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T11:06:31.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 -- 172.21.15.78:0/1456780692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe8102290 msgr2=0x7fcfe8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 --2- 172.21.15.78:0/1456780692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe8102290 0x7fcfe8108b20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009ca0 tx=0x7fcfe002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 -- 172.21.15.78:0/1456780692 shutdown_connections 2024-03-16T11:06:31.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 --2- 172.21.15.78:0/1456780692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe8102290 0x7fcfe8108b20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 --2- 172.21.15.78:0/1456780692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe81018d0 0x7fcfe8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 --2- 172.21.15.78:0/1456780692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe81006e0 0x7fcfe8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 -- 172.21.15.78:0/1456780692 >> 172.21.15.78:0/1456780692 conn(0x7fcfe80fbc70 msgr2=0x7fcfe80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 -- 172.21.15.78:0/1456780692 shutdown_connections 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 -- 172.21.15.78:0/1456780692 wait complete. 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 Processor -- start 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.360+0000 7fcfefec6640 1 -- start start 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe81006e0 0x7fcfe81968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe81018d0 0x7fcfe8196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe8102290 0x7fcfe819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe810c210 con 0x7fcfe81018d0 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe810c090 con 0x7fcfe8102290 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe810c390 con 0x7fcfe81006e0 2024-03-16T11:06:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe81018d0 0x7fcfe8196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe81018d0 0x7fcfe8196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40824/0 (socket says 172.21.15.78:40824) 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 -- 172.21.15.78:0/3891419974 learned_addr learned my addr 172.21.15.78:0/3891419974 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfedc62640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe81006e0 0x7fcfe81968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfee463640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe8102290 0x7fcfe819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 -- 172.21.15.78:0/3891419974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe81006e0 msgr2=0x7fcfe81968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe81006e0 0x7fcfe81968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 -- 172.21.15.78:0/3891419974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe8102290 msgr2=0x7fcfe819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe8102290 0x7fcfe819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 -- 172.21.15.78:0/3891419974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe819e510 con 0x7fcfe81018d0 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfed461640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe81018d0 0x7fcfe8196e30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400e9f0 tx=0x7fcfe400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfdaffd640 1 -- 172.21.15.78:0/3891419974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe400cd80 con 0x7fcfe81018d0 2024-03-16T11:06:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfdaffd640 1 -- 172.21.15.78:0/3891419974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe40054d0 con 0x7fcfe81018d0 2024-03-16T11:06:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe819e800 con 0x7fcfe81018d0 2024-03-16T11:06:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfdaffd640 1 -- 172.21.15.78:0/3891419974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4010490 con 0x7fcfe81018d0 2024-03-16T11:06:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.361+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe819ed00 con 0x7fcfe81018d0 2024-03-16T11:06:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.363+0000 7fcfdaffd640 1 -- 172.21.15.78:0/3891419974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fcfe4005020 con 0x7fcfe81018d0 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 Processor -- start 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 -- start start 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499793ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfdaffd640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfd40a2890 0x7fcfd40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499793fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997942840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56499789f490 con 0x56499793a850 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564997847c90 con 0x5649979403b0 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcff2b91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56499789b530 con 0x56499793b240 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499793ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfedc62640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfd40a2890 0x7fcfd40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499793ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40838/0 (socket says 172.21.15.78:40838) 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfed461640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499793fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 -- 172.21.15.78:0/695257483 learned_addr learned my addr 172.21.15.78:0/695257483 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997942840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfdaffd640 1 -- 172.21.15.78:0/3891419974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcfe40cef90 con 0x7fcfe81018d0 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 -- 172.21.15.78:0/695257483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 msgr2=0x56499793fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499793fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 -- 172.21.15.78:0/695257483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 msgr2=0x564997942840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997942840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.364+0000 7fcfee463640 1 -- 172.21.15.78:0/695257483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564997902500 con 0x56499793a850 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.365+0000 7fcfedc62640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfd40a2890 0x7fcfd40a4d50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0095f0 tx=0x7fcfdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.365+0000 7fcfee463640 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499793ac70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0032420 tx=0x7fcfe0032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ef219d7355a808 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.365+0000 7fcfd3fff640 1 -- 172.21.15.78:0/695257483 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0048070 con 0x56499793a850 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.365+0000 7fcfd3fff640 1 -- 172.21.15.78:0/695257483 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe0003970 con 0x56499793a850 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.365+0000 7fcfd3fff640 1 -- 172.21.15.78:0/695257483 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0039040 con 0x56499793a850 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.365+0000 7fcff2b91740 1 -- 172.21.15.78:0/695257483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 msgr2=0x56499793ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.365+0000 7fcff2b91740 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499793ac70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0032420 tx=0x7fcfe0032340 comp rx=0 tx=0).stop 2024-03-16T11:06:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- 172.21.15.78:0/695257483 shutdown_connections 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997942840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499793fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 --2- 172.21.15.78:0/695257483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499793ac70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- 172.21.15.78:0/695257483 >> 172.21.15.78:0/695257483 conn(0x564997935dc0 msgr2=0x5649979361c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- 172.21.15.78:0/695257483 shutdown_connections 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- 172.21.15.78:0/695257483 wait complete. 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- start start 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 Processor -- start 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56499793a850 0x564997946f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x5649979474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649979403b0 0x56499794e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x564997945510 con 0x56499793b240 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56499785b230 con 0x56499793a850 2024-03-16T11:06:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564997945690 con 0x5649979403b0 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcff2b91740 10 client.0 _ll_register_callbacks cb 0x564997897010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.366+0000 7fcfed461640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x5649979474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x5649979474b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40840/0 (socket says 172.21.15.78:40840) 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfedc62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56499793a850 0x564997946f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 -- 172.21.15.78:0/2494271306 learned_addr learned my addr 172.21.15.78:0/2494271306 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfee463640 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649979403b0 0x56499794e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 -- 172.21.15.78:0/2494271306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649979403b0 msgr2=0x56499794e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649979403b0 0x56499794e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 -- 172.21.15.78:0/2494271306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56499793a850 msgr2=0x564997946f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56499793a850 0x564997946f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 -- 172.21.15.78:0/2494271306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499786a100 con 0x56499793b240 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfed461640 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x5649979474b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40075f0 tx=0x7fcfe409b130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c75b32c74b5ef35 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfd17fa640 1 -- 172.21.15.78:0/2494271306 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4005310 con 0x56499793b240 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfd17fa640 1 -- 172.21.15.78:0/2494271306 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcfe40ccb90 con 0x56499793b240 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfe0003970 con 0x56499793b240 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfd17fa640 1 -- 172.21.15.78:0/2494271306 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcfe4005680 con 0x56499793b240 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcff2b91740 10 client.15561 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5649975b33f0 con 0x56499793b240 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfd17fa640 1 -- 172.21.15.78:0/2494271306 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe0003970 con 0x56499793b240 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcfd17fa640 1 -- 172.21.15.78:0/2494271306 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcfe4086070 con 0x56499793b240 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcff2b91740 10 client.15561 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcff2b91740 10 client.15561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcff2b91740 20 client.15561 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.367+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x564997952cd0 0x564997955190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.368+0000 7fcff2b91740 4 client.15561 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.368+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649978ac9d0 con 0x564997952cd0 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.368+0000 7fcfedc62640 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x564997952cd0 0x564997955190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.368+0000 7fcfd17fa640 10 client.15561 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.368+0000 7fcfedc62640 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x564997952cd0 0x564997955190 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22667e96e4684196 server_cookie=eabb1ad418d40031 in_seq=0 out_seq=0 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcfd17fa640 1 -- 172.21.15.78:0/2494271306 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649978ac9d0 con 0x564997952cd0 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcfd17fa640 10 client.15561 handle_command_reply: tid=0 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 client.15561 shutdown 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 msgr2=0x5649979474b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x5649979474b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40075f0 tx=0x7fcfe409b130 comp rx=0 tx=0).stop 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 shutdown_connections 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x564997952cd0 0x564997955190 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649979403b0 0x56499794e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x5649979474b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2494271306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56499793a850 0x564997946f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcfd17fa640 0 client.15561 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 >> 172.21.15.78:0/2494271306 conn(0x564997935dc0 msgr2=0x564997934fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 shutdown_connections 2024-03-16T11:06:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 1 -- 172.21.15.78:0/2494271306 wait complete. 2024-03-16T11:06:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.369+0000 7fcff2b91740 20 client.15561 trim_cache size 0 max 16384 2024-03-16T11:06:31.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 Processor -- start 2024-03-16T11:06:31.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 -- start start 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793a850 0x564997939730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x56499794e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997959a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81a92d0 con 0x56499793b240 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81a6ba0 con 0x5649979403b0 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcff2b91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81a49d0 con 0x56499793a850 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x56499794e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x56499794e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40846/0 (socket says 172.21.15.78:40846) 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 -- 172.21.15.78:0/2648882016 learned_addr learned my addr 172.21.15.78:0/2648882016 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfee463640 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997959a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfedc62640 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793a850 0x564997939730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 -- 172.21.15.78:0/2648882016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793a850 msgr2=0x564997939730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793a850 0x564997939730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 -- 172.21.15.78:0/2648882016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 msgr2=0x564997959a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997959a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 -- 172.21.15.78:0/2648882016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56499767d7d0 con 0x56499793b240 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfed461640 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x56499794e4b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400c6a0 tx=0x7fcfe409b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd1927e488d7f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.382+0000 7fcfd37fe640 1 -- 172.21.15.78:0/2648882016 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4097a70 con 0x56499793b240 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcfd37fe640 1 -- 172.21.15.78:0/2648882016 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe40b4070 con 0x56499793b240 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcfd37fe640 1 -- 172.21.15.78:0/2648882016 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe40a0840 con 0x56499793b240 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 -- 172.21.15.78:0/2648882016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 msgr2=0x56499794e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x56499794e4b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400c6a0 tx=0x7fcfe409b260 comp rx=0 tx=0).stop 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 -- 172.21.15.78:0/2648882016 shutdown_connections 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x564997959a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793b240 0x56499794e4b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 --2- 172.21.15.78:0/2648882016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793a850 0x564997939730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 -- 172.21.15.78:0/2648882016 >> 172.21.15.78:0/2648882016 conn(0x564997935dc0 msgr2=0x564997939020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 -- 172.21.15.78:0/2648882016 shutdown_connections 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 -- 172.21.15.78:0/2648882016 wait complete. 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 -- start start 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.383+0000 7fcff2b91740 1 Processor -- start 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499794b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499794be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x56499794ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81a5060 con 0x56499793a850 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81a64d0 con 0x5649979403b0 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81a5730 con 0x56499793b240 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 10 client.0 _ll_register_callbacks cb 0x7fcfe400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499794b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499794b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40848/0 (socket says 172.21.15.78:40848) 2024-03-16T11:06:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 -- 172.21.15.78:0/1240890470 learned_addr learned my addr 172.21.15.78:0/1240890470 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfee463640 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x56499794ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfed461640 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499794be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 -- 172.21.15.78:0/1240890470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 msgr2=0x56499794be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499794be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 -- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 msgr2=0x56499794ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x56499794ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 -- 172.21.15.78:0/1240890470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499794d430 con 0x56499793a850 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfedc62640 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499794b8f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0099c0 tx=0x7fcfdc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f52ca5ecb897b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfd17fa640 1 -- 172.21.15.78:0/1240890470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc0271b0 con 0x56499793a850 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcfd17fa640 1 -- 172.21.15.78:0/1240890470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcfdc027350 con 0x56499793a850 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.384+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfe40b4070 con 0x56499793a850 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfd17fa640 1 -- 172.21.15.78:0/1240890470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcfdc027550 con 0x56499793a850 2024-03-16T11:06:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 10 client.15567 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56499794e800 con 0x56499793a850 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfd17fa640 1 -- 172.21.15.78:0/1240890470 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe40b4070 con 0x56499793a850 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfd17fa640 1 -- 172.21.15.78:0/1240890470 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcfdc02fc40 con 0x56499793a850 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 10 client.15567 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 10 client.15567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 20 client.15567 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5649979b98f0 0x5649979bbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 4 client.15567 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649978a6d20 con 0x5649979b98f0 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfed461640 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5649979b98f0 0x5649979bbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfd17fa640 10 client.15567 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfed461640 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5649979b98f0 0x5649979bbd10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ba759c0614f3a3a server_cookie=217b8cfae6db958d in_seq=0 out_seq=0 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfd17fa640 1 -- 172.21.15.78:0/1240890470 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649978a6d20 con 0x5649979b98f0 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.385+0000 7fcfd17fa640 10 client.15567 handle_command_reply: tid=0 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 client.15567 shutdown 2024-03-16T11:06:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 msgr2=0x56499794b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499794b8f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0099c0 tx=0x7fcfdc004730 comp rx=0 tx=0).stop 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 shutdown_connections 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5649979b98f0 0x5649979bbd10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649979403b0 0x56499794ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfd17fa640 0 client.15567 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499793b240 0x56499794be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 --2- 172.21.15.78:0/1240890470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56499793a850 0x56499794b8f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 >> 172.21.15.78:0/1240890470 conn(0x564997935dc0 msgr2=0x5649979398b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 shutdown_connections 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 1 -- 172.21.15.78:0/1240890470 wait complete. 2024-03-16T11:06:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcff2b91740 20 client.15567 trim_cache size 0 max 16384 2024-03-16T11:06:31.395 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:06:31.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfd40a2890 msgr2=0x7fcfd40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfd40a2890 0x7fcfd40a4d50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0095f0 tx=0x7fcfdc005f90 comp rx=0 tx=0).stop 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe81018d0 msgr2=0x7fcfe8196e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe81018d0 0x7fcfe8196e30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400e9f0 tx=0x7fcfe400edf0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 shutdown_connections 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe8102290 0x7fcfe819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe81018d0 0x7fcfe8196e30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfd40a2890 0x7fcfd40a4d50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 --2- 172.21.15.78:0/3891419974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfe81006e0 0x7fcfe81968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 >> 172.21.15.78:0/3891419974 conn(0x7fcfe80fbc70 msgr2=0x7fcfe8104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.386+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 shutdown_connections 2024-03-16T11:06:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:31.387+0000 7fcfefec6640 1 -- 172.21.15.78:0/3891419974 wait complete. 2024-03-16T11:06:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:31 smithi098 ceph-mon[35848]: pgmap v364: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-16T11:06:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:31 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/485659319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:06:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:31 smithi153 ceph-mon[35521]: pgmap v364: 97 pgs: 97 active+clean; 660 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-16T11:06:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:31 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/485659319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:06:32.398 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T11:06:32.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:06:32.398 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 -- 172.21.15.78:0/930624270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1590100700 msgr2=0x7f1590100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 --2- 172.21.15.78:0/930624270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1590100700 0x7f1590100b20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1584009900 tx=0x7f158402bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 -- 172.21.15.78:0/930624270 shutdown_connections 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 --2- 172.21.15.78:0/930624270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15901022b0 0x7f1590108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 --2- 172.21.15.78:0/930624270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15901018f0 0x7f1590101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 --2- 172.21.15.78:0/930624270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1590100700 0x7f1590100b20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 -- 172.21.15.78:0/930624270 >> 172.21.15.78:0/930624270 conn(0x7f15900fbc90 msgr2=0x7f15900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 -- 172.21.15.78:0/930624270 shutdown_connections 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 -- 172.21.15.78:0/930624270 wait complete. 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 Processor -- start 2024-03-16T11:06:32.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.518+0000 7f159631b640 1 -- start start 2024-03-16T11:06:32.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f159631b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590100700 0x7f1590197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f159631b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15901018f0 0x7f1590197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f159631b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15901022b0 0x7f159019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f159631b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f159010c230 con 0x7f15901018f0 2024-03-16T11:06:32.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f159631b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f159010c0b0 con 0x7f1590100700 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f159631b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f159010c3b0 con 0x7f15901022b0 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15901018f0 0x7f1590197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15901018f0 0x7f1590197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40870/0 (socket says 172.21.15.78:40870) 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 -- 172.21.15.78:0/3956170963 learned_addr learned my addr 172.21.15.78:0/3956170963 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158ffff640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590100700 0x7f1590197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f15948b8640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15901022b0 0x7f159019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 -- 172.21.15.78:0/3956170963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15901022b0 msgr2=0x7f159019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15901022b0 0x7f159019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 -- 172.21.15.78:0/3956170963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590100700 msgr2=0x7f1590197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590100700 0x7f1590197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 -- 172.21.15.78:0/3956170963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159019f650 con 0x7f15901018f0 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158ffff640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590100700 0x7f1590197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.519+0000 7f158f7fe640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15901018f0 0x7f1590197f70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f158000e9f0 tx=0x7f158000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.520+0000 7f158d7fa640 1 -- 172.21.15.78:0/3956170963 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158000cd80 con 0x7f15901018f0 2024-03-16T11:06:32.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.520+0000 7f158d7fa640 1 -- 172.21.15.78:0/3956170963 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15800108e0 con 0x7f15901018f0 2024-03-16T11:06:32.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.520+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159019f940 con 0x7f15901018f0 2024-03-16T11:06:32.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.520+0000 7f158d7fa640 1 -- 172.21.15.78:0/3956170963 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580018920 con 0x7f15901018f0 2024-03-16T11:06:32.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.520+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f159019fe90 con 0x7f15901018f0 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.522+0000 7f158d7fa640 1 -- 172.21.15.78:0/3956170963 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f1580010a90 con 0x7f15901018f0 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.522+0000 7f1598fe3740 1 Processor -- start 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.522+0000 7f158d7fa640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15780a2890 0x7f15780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.522+0000 7f1598fe3740 1 -- start start 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b997211c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f15948b8640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15780a2890 0x7f15780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99721790 0x557b99726290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158d7fa640 1 -- 172.21.15.78:0/3956170963 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f15800ceee0 con 0x7f15901018f0 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99726900 0x557b99728d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f1598fe3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557b996810d0 con 0x557b99726900 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f1598fe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557b99629c70 con 0x557b99720da0 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f1598fe3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557b9967d170 con 0x557b99721790 2024-03-16T11:06:32.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b997211c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f15948b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99726900 0x557b99728d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b997211c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:46634/0 (socket says 172.21.15.78:46634) 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 -- 172.21.15.78:0/3581769896 learned_addr learned my addr 172.21.15.78:0/3581769896 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99721790 0x557b99726290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 -- 172.21.15.78:0/3581769896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99721790 msgr2=0x557b99726290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99721790 0x557b99726290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 -- 172.21.15.78:0/3581769896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99726900 msgr2=0x557b99728d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99726900 0x557b99728d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 -- 172.21.15.78:0/3581769896 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9945d7e0 con 0x557b99720da0 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f15948b8640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15780a2890 0x7f15780a4d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1588005fd0 tx=0x7f1588013b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158f7fe640 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99721790 0x557b99726290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f158ffff640 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b997211c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1584009900 tx=0x7f15840095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a88672df83bc0d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.523+0000 7f15767fc640 1 -- 172.21.15.78:0/3581769896 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1584045020 con 0x557b99720da0 2024-03-16T11:06:32.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f15767fc640 1 -- 172.21.15.78:0/3581769896 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f158402cc90 con 0x557b99720da0 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f15767fc640 1 -- 172.21.15.78:0/3581769896 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1584004920 con 0x557b99720da0 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 -- 172.21.15.78:0/3581769896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 msgr2=0x557b997211c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b997211c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1584009900 tx=0x7f15840095f0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 -- 172.21.15.78:0/3581769896 shutdown_connections 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99726900 0x557b99728d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99721790 0x557b99726290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3581769896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b997211c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 -- 172.21.15.78:0/3581769896 >> 172.21.15.78:0/3581769896 conn(0x557b996e4c50 msgr2=0x557b99718300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 -- 172.21.15.78:0/3581769896 shutdown_connections 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.524+0000 7f1598fe3740 1 -- 172.21.15.78:0/3581769896 wait complete. 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 -- start start 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 Processor -- start 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b9972b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b9972b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557b99729850 con 0x557b99721790 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557b9963d0d0 con 0x557b99726900 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557b997299d0 con 0x557b99720da0 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f1598fe3740 10 client.0 _ll_register_callbacks cb 0x557b99606760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b9972b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b9972b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40896/0 (socket says 172.21.15.78:40896) 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158f7fe640 1 -- 172.21.15.78:0/4260375879 learned_addr learned my addr 172.21.15.78:0/4260375879 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158ffff640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b9972b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f15948b8640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158ffff640 1 -- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 msgr2=0x557b997327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158ffff640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158ffff640 1 -- 172.21.15.78:0/4260375879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 msgr2=0x557b9972b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158ffff640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b9972b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158ffff640 1 -- 172.21.15.78:0/4260375879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b994bdf70 con 0x557b99720da0 2024-03-16T11:06:32.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.525+0000 7f158f7fe640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b9972b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f158ffff640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b9972b220 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1584000c00 tx=0x7f158402c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa1622cdfe383534 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f156f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f156f7fe640 1 -- 172.21.15.78:0/4260375879 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1584050020 con 0x557b99720da0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f156f7fe640 1 -- 172.21.15.78:0/4260375879 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1584003bc0 con 0x557b99720da0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f158402cc90 con 0x557b99720da0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f156f7fe640 1 -- 172.21.15.78:0/4260375879 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1584045040 con 0x557b99720da0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f1598fe3740 10 client.25399 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557b99355170 con 0x557b99720da0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f156f7fe640 1 -- 172.21.15.78:0/4260375879 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f158402cc90 con 0x557b99720da0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.526+0000 7f156f7fe640 1 -- 172.21.15.78:0/4260375879 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f158404c460 con 0x557b99720da0 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f1598fe3740 10 client.25399 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f1598fe3740 10 client.25399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f1598fe3740 20 client.25399 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b99736900 0x557b99738cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f1598fe3740 4 client.25399 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b9968e610 con 0x557b99736900 2024-03-16T11:06:32.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f158f7fe640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b99736900 0x557b99738cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f158f7fe640 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b99736900 0x557b99738cf0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a3204204a93466c server_cookie=9588b0c1abf8c302 in_seq=0 out_seq=0 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.527+0000 7f156f7fe640 10 client.25399 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f156f7fe640 1 -- 172.21.15.78:0/4260375879 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b9968e610 con 0x557b99736900 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f156f7fe640 10 client.25399 handle_command_reply: tid=0 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 client.25399 shutdown 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 msgr2=0x557b9972b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b9972b220 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1584000c00 tx=0x7f158402c330 comp rx=0 tx=0).stop 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 shutdown_connections 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b99736900 0x557b99738cf0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b9972b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f156f7fe640 0 client.25399 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4260375879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b9972b220 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 >> 172.21.15.78:0/4260375879 conn(0x557b996e4c50 msgr2=0x557b997174e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 shutdown_connections 2024-03-16T11:06:32.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 1 -- 172.21.15.78:0/4260375879 wait complete. 2024-03-16T11:06:32.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.528+0000 7f1598fe3740 20 client.25399 trim_cache size 0 max 16384 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 Processor -- start 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 -- start start 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b9973e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b99732760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99726900 0x557b99723c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f15901a9210 con 0x557b99721790 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15901a6ae0 con 0x557b99720da0 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.545+0000 7f1598fe3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15901a4910 con 0x557b99726900 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b99732760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b9973e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b99732760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40900/0 (socket says 172.21.15.78:40900) 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 -- 172.21.15.78:0/3509112285 learned_addr learned my addr 172.21.15.78:0/3509112285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f15948b8640 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99726900 0x557b99723c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 -- 172.21.15.78:0/3509112285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99726900 msgr2=0x557b99723c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99726900 0x557b99723c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 -- 172.21.15.78:0/3509112285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 msgr2=0x557b9973e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b9973e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 -- 172.21.15.78:0/3509112285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b997392b0 con 0x557b99721790 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158ffff640 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b9973e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f158f7fe640 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b99732760 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f15800981a0 tx=0x7f1580009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b97146c492c9d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f1575ffb640 1 -- 172.21.15.78:0/3509112285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15800ccae0 con 0x557b99721790 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f1575ffb640 1 -- 172.21.15.78:0/3509112285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1580097e40 con 0x557b99721790 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.546+0000 7f1575ffb640 1 -- 172.21.15.78:0/3509112285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15800afd60 con 0x557b99721790 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 -- 172.21.15.78:0/3509112285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 msgr2=0x557b99732760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b99732760 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f15800981a0 tx=0x7f1580009310 comp rx=0 tx=0).stop 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 -- 172.21.15.78:0/3509112285 shutdown_connections 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99726900 0x557b99723c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b99732760 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 --2- 172.21.15.78:0/3509112285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99720da0 0x557b9973e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 -- 172.21.15.78:0/3509112285 >> 172.21.15.78:0/3509112285 conn(0x557b996e4c50 msgr2=0x557b997174e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 -- 172.21.15.78:0/3509112285 shutdown_connections 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 -- 172.21.15.78:0/3509112285 wait complete. 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 -- start start 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.547+0000 7f1598fe3740 1 Processor -- start 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b997302a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b997307e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f15901a4fa0 con 0x557b99721790 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15901a6410 con 0x557b99726900 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15901a5670 con 0x557b99720da0 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 10 client.0 _ll_register_callbacks cb 0x7f1588009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b997307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b997307e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40908/0 (socket says 172.21.15.78:40908) 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 -- 172.21.15.78:0/4269444482 learned_addr learned my addr 172.21.15.78:0/4269444482 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b997302a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f15948b8640 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 -- 172.21.15.78:0/4269444482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 msgr2=0x557b997302a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b997302a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 -- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 msgr2=0x557b997317f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997317f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 -- 172.21.15.78:0/4269444482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b99731de0 con 0x557b99721790 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f158f7fe640 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b997307e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f158009f960 tx=0x7f15800ccd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3265137a4a6499e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f156f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f156f7fe640 1 -- 172.21.15.78:0/4269444482 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158009ac20 con 0x557b99721790 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f156f7fe640 1 -- 172.21.15.78:0/4269444482 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1580010640 con 0x557b99721790 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1580097e40 con 0x557b99721790 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f156f7fe640 1 -- 172.21.15.78:0/4269444482 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f15800a6070 con 0x557b99721790 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.548+0000 7f1598fe3740 10 client.15585 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:32.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557b9979cea0 con 0x557b99721790 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f156f7fe640 1 -- 172.21.15.78:0/4269444482 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1580097e40 con 0x557b99721790 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f156f7fe640 1 -- 172.21.15.78:0/4269444482 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1580097520 con 0x557b99721790 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f1598fe3740 10 client.15585 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f1598fe3740 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f1598fe3740 20 client.15585 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b9979d810 0x557b9979fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f1598fe3740 4 client.15585 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557b99688960 con 0x557b9979d810 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f158ffff640 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b9979d810 0x557b9979fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f156f7fe640 10 client.15585 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.549+0000 7f158ffff640 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b9979d810 0x557b9979fc30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1317dc963ac71c3e server_cookie=f5ce215108a5e2af in_seq=0 out_seq=0 2024-03-16T11:06:32.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.550+0000 7f156f7fe640 1 -- 172.21.15.78:0/4269444482 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557b99688960 con 0x557b9979d810 2024-03-16T11:06:32.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.550+0000 7f156f7fe640 10 client.15585 handle_command_reply: tid=0 2024-03-16T11:06:32.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.550+0000 7f1598fe3740 1 client.15585 shutdown 2024-03-16T11:06:32.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 msgr2=0x557b997307e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b997307e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f158009f960 tx=0x7f15800ccd10 comp rx=0 tx=0).stop 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 shutdown_connections 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x557b9979d810 0x557b9979fc30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557b99726900 0x557b997317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b99721790 0x557b997307e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f156f7fe640 0 client.15585 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 --2- 172.21.15.78:0/4269444482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b99720da0 0x557b997302a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 >> 172.21.15.78:0/4269444482 conn(0x557b996e4c50 msgr2=0x557b9973d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 shutdown_connections 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 1 -- 172.21.15.78:0/4269444482 wait complete. 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f1598fe3740 20 client.15585 trim_cache size 0 max 16384 2024-03-16T11:06:32.556 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:06:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-16T11:06:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "0979cb6b-2d19-4622-8f9b-9566da68c747", 2024-03-16T11:06:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-16T11:06:32.557 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:06:32.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15780a2890 msgr2=0x7f15780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15780a2890 0x7f15780a4d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1588005fd0 tx=0x7f1588013b60 comp rx=0 tx=0).stop 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15901018f0 msgr2=0x7f1590197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15901018f0 0x7f1590197f70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f158000e9f0 tx=0x7f158000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 shutdown_connections 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f15780a2890 0x7f15780a4d50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15901022b0 0x7f159019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15901018f0 0x7f1590197f70 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 --2- 172.21.15.78:0/3956170963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590100700 0x7f1590197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 >> 172.21.15.78:0/3956170963 conn(0x7f15900fbc90 msgr2=0x7f1590069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 shutdown_connections 2024-03-16T11:06:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.551+0000 7f159631b640 1 -- 172.21.15.78:0/3956170963 wait complete. 2024-03-16T11:06:32.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:06:32.563 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 -- 172.21.15.78:0/4079069035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b201018f0 msgr2=0x7f8b20101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 --2- 172.21.15.78:0/4079069035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b201018f0 0x7f8b20101d70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009900 tx=0x7f8b1402bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 -- 172.21.15.78:0/4079069035 shutdown_connections 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 --2- 172.21.15.78:0/4079069035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b201022b0 0x7f8b20108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 --2- 172.21.15.78:0/4079069035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b201018f0 0x7f8b20101d70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 --2- 172.21.15.78:0/4079069035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b20100700 0x7f8b20100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 -- 172.21.15.78:0/4079069035 >> 172.21.15.78:0/4079069035 conn(0x7f8b200fbc90 msgr2=0x7f8b200fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 -- 172.21.15.78:0/4079069035 shutdown_connections 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 -- 172.21.15.78:0/4079069035 wait complete. 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.667+0000 7f8b25377640 1 Processor -- start 2024-03-16T11:06:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 -- start start 2024-03-16T11:06:32.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b20100700 0x7f8b2006a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b201018f0 0x7f8b2019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b201022b0 0x7f8b2019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010c230 con 0x7f8b20100700 2024-03-16T11:06:32.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010c0b0 con 0x7f8b201018f0 2024-03-16T11:06:32.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010c3b0 con 0x7f8b201022b0 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b20100700 0x7f8b2006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b20100700 0x7f8b2006a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40924/0 (socket says 172.21.15.78:40924) 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 -- 172.21.15.78:0/2438037601 learned_addr learned my addr 172.21.15.78:0/2438037601 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b201018f0 0x7f8b2019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b201022b0 0x7f8b2019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 -- 172.21.15.78:0/2438037601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b201022b0 msgr2=0x7f8b2019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b201022b0 0x7f8b2019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 -- 172.21.15.78:0/2438037601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b201018f0 msgr2=0x7f8b2019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b201018f0 0x7f8b2019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 -- 172.21.15.78:0/2438037601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2019f6b0 con 0x7f8b20100700 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b201018f0 0x7f8b2019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b1effd640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b20100700 0x7f8b2006a980 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000e9c0 tx=0x7f8b1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b07fff640 1 -- 172.21.15.78:0/2438037601 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1000cd80 con 0x7f8b20100700 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.668+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2019f9a0 con 0x7f8b20100700 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.669+0000 7f8b07fff640 1 -- 172.21.15.78:0/2438037601 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b100035f0 con 0x7f8b20100700 2024-03-16T11:06:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.669+0000 7f8b07fff640 1 -- 172.21.15.78:0/2438037601 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b10010690 con 0x7f8b20100700 2024-03-16T11:06:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.669+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b2019fec0 con 0x7f8b20100700 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.671+0000 7f8b07fff640 1 -- 172.21.15.78:0/2438037601 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f8b100037a0 con 0x7f8b20100700 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.671+0000 7f8b27f67740 1 Processor -- start 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.671+0000 7f8b27f67740 1 -- start start 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3af170 0x560f9a3af590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3b7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b27f67740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560f9a314500 con 0x560f9a3afb60 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b27f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560f9a2bcbe0 con 0x560f9a3b4cd0 2024-03-16T11:06:32.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b27f67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560f9a3105a0 con 0x560f9a3af170 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40928/0 (socket says 172.21.15.78:40928) 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 -- 172.21.15.78:0/3873779544 learned_addr learned my addr 172.21.15.78:0/3873779544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3af170 0x560f9a3af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b07fff640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b080a2890 0x7f8b080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 -- 172.21.15.78:0/3873779544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3af170 msgr2=0x560f9a3af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3af170 0x560f9a3af590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 -- 172.21.15.78:0/3873779544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 msgr2=0x560f9a3b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 -- 172.21.15.78:0/3873779544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f9a2d04b0 con 0x560f9a3afb60 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b080a2890 0x7f8b080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b07fff640 1 -- 172.21.15.78:0/2438037601 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8b100cf140 con 0x7f8b20100700 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1effd640 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b4660 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8b100cca30 tx=0x7f8b10066740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fc8d29f73fa11f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b057fa640 1 -- 172.21.15.78:0/3873779544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1009b750 con 0x560f9a3afb60 2024-03-16T11:06:32.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b057fa640 1 -- 172.21.15.78:0/3873779544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b10003b10 con 0x560f9a3afb60 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b057fa640 1 -- 172.21.15.78:0/3873779544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b100d2d60 con 0x560f9a3afb60 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.672+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b080a2890 0x7f8b080a4d50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009fd0 tx=0x7f8b18004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 -- 172.21.15.78:0/3873779544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 msgr2=0x560f9a3b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b4660 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8b100cca30 tx=0x7f8b10066740 comp rx=0 tx=0).stop 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 -- 172.21.15.78:0/3873779544 shutdown_connections 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b4660 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3873779544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3af170 0x560f9a3af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 -- 172.21.15.78:0/3873779544 >> 172.21.15.78:0/3873779544 conn(0x560f9a3aa6e0 msgr2=0x560f9a3aaae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 -- 172.21.15.78:0/3873779544 shutdown_connections 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 -- 172.21.15.78:0/3873779544 wait complete. 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.673+0000 7f8b27f67740 1 -- start start 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 1 Processor -- start 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3af170 0x560f9a3b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560f9a3b7c20 con 0x560f9a3afb60 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560f9a2d0160 con 0x560f9a3af170 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560f9a3b7da0 con 0x560f9a3b4cd0 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b27f67740 10 client.0 _ll_register_callbacks cb 0x560f9a2b4150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40932/0 (socket says 172.21.15.78:40932) 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 -- 172.21.15.78:0/2458848063 learned_addr learned my addr 172.21.15.78:0/2458848063 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1effd640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3af170 0x560f9a3b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 -- 172.21.15.78:0/2458848063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 msgr2=0x560f9a3c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 -- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3af170 msgr2=0x560f9a3b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3af170 0x560f9a3b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 -- 172.21.15.78:0/2458848063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f9a0f27d0 con 0x560f9a3afb60 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1effd640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3af170 0x560f9a3b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:32.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b9b40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8b1402c170 tx=0x7f8b14005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81122d8c1c921e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.674+0000 7f8b02ff5640 1 -- 172.21.15.78:0/2458848063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14005d50 con 0x560f9a3afb60 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b02ff5640 1 -- 172.21.15.78:0/2458848063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8b140046c0 con 0x560f9a3afb60 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560f9a321a40 con 0x560f9a3afb60 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b02ff5640 1 -- 172.21.15.78:0/2458848063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8b1402cc90 con 0x560f9a3afb60 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 10 client.15597 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560f9a0283f0 con 0x560f9a3afb60 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b02ff5640 1 -- 172.21.15.78:0/2458848063 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560f9a321a40 con 0x560f9a3afb60 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b02ff5640 1 -- 172.21.15.78:0/2458848063 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8b14003a70 con 0x560f9a3afb60 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 10 client.15597 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 20 client.15597 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a3c5100 0x560f9a3c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 4 client.15597 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:32.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f9a31bd90 con 0x560f9a3c5100 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.675+0000 7f8b1effd640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a3c5100 0x560f9a3c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b1effd640 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a3c5100 0x560f9a3c75a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=540da80348edb854 server_cookie=18b131f54fc96c72 in_seq=0 out_seq=0 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b02ff5640 10 client.15597 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b02ff5640 1 -- 172.21.15.78:0/2458848063 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f9a31bd90 con 0x560f9a3c5100 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b02ff5640 10 client.15597 handle_command_reply: tid=0 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 client.15597 shutdown 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 msgr2=0x560f9a3b9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b9b40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8b1402c170 tx=0x7f8b14005250 comp rx=0 tx=0).stop 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 shutdown_connections 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a3c5100 0x560f9a3c75a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b02ff5640 0 client.15597 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3afb60 0x560f9a3b9b40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 --2- 172.21.15.78:0/2458848063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3af170 0x560f9a3b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 >> 172.21.15.78:0/2458848063 conn(0x560f9a3aa6e0 msgr2=0x560f9a3a9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.676+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 shutdown_connections 2024-03-16T11:06:32.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.677+0000 7f8b27f67740 1 -- 172.21.15.78:0/2458848063 wait complete. 2024-03-16T11:06:32.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.677+0000 7f8b27f67740 20 client.15597 trim_cache size 0 max 16384 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.693+0000 7f8b27f67740 1 Processor -- start 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.693+0000 7f8b27f67740 1 -- start start 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3a9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3afb60 0x560f9a3c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b27f67740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8b201ab4e0 con 0x560f9a3af170 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b27f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b201a8db0 con 0x560f9a3b4cd0 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b27f67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b201a6be0 con 0x560f9a3afb60 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3a9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40934/0 (socket says 172.21.15.78:40934) 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 -- 172.21.15.78:0/3350267367 learned_addr learned my addr 172.21.15.78:0/3350267367 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3afb60 0x560f9a3c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 -- 172.21.15.78:0/3350267367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3afb60 msgr2=0x560f9a3c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3afb60 0x560f9a3c0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 -- 172.21.15.78:0/3350267367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 msgr2=0x560f9a3a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 -- 172.21.15.78:0/3350267367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f99fe80f0 con 0x560f9a3af170 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b1effd640 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3a9070 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000c6a0 tx=0x7f8b10003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6ffacfa3bac847 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b04ff9640 1 -- 172.21.15.78:0/3350267367 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1009b2e0 con 0x560f9a3af170 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b04ff9640 1 -- 172.21.15.78:0/3350267367 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b1009c080 con 0x560f9a3af170 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.694+0000 7f8b04ff9640 1 -- 172.21.15.78:0/3350267367 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1009b4c0 con 0x560f9a3af170 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- 172.21.15.78:0/3350267367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 msgr2=0x560f9a3a9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3a9070 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000c6a0 tx=0x7f8b10003e70 comp rx=0 tx=0).stop 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- 172.21.15.78:0/3350267367 shutdown_connections 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3b4cd0 0x560f9a3a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3afb60 0x560f9a3c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 --2- 172.21.15.78:0/3350267367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3a9070 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- 172.21.15.78:0/3350267367 >> 172.21.15.78:0/3350267367 conn(0x560f9a3aa6e0 msgr2=0x560f9a3a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- 172.21.15.78:0/3350267367 shutdown_connections 2024-03-16T11:06:32.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- 172.21.15.78:0/3350267367 wait complete. 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- start start 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 Processor -- start 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3be5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3afb60 0x560f9a3beb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8b201a7270 con 0x560f9a3af170 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b201a86e0 con 0x560f9a3afb60 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b201a7940 con 0x560f9a3b4cd0 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b27f67740 10 client.0 _ll_register_callbacks cb 0x7f8b14009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3be5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b1effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3be5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40944/0 (socket says 172.21.15.78:40944) 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.695+0000 7f8b1effd640 1 -- 172.21.15.78:0/1654271050 learned_addr learned my addr 172.21.15.78:0/1654271050 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3afb60 0x560f9a3beb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1f7fe640 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1effd640 1 -- 172.21.15.78:0/1654271050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 msgr2=0x560f9a3bfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1effd640 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3bfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1effd640 1 -- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3afb60 msgr2=0x560f9a3beb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1effd640 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3afb60 0x560f9a3beb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1effd640 1 -- 172.21.15.78:0/1654271050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f9a2db9d0 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b1effd640 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3be5d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8b100cd070 tx=0x7f8b100cca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685030222fd10312 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b02ff5640 1 -- 172.21.15.78:0/1654271050 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1009b4c0 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b02ff5640 1 -- 172.21.15.78:0/1654271050 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8b100ccd40 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b1009c080 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b02ff5640 1 -- 172.21.15.78:0/1654271050 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8b1009b2e0 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b27f67740 10 client.15603 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560f9a42b5c0 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b02ff5640 1 -- 172.21.15.78:0/1654271050 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b1009c080 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.696+0000 7f8b02ff5640 1 -- 172.21.15.78:0/1654271050 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8b100a3c90 con 0x560f9a3af170 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 10 client.15603 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:32.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 20 client.15603 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a42ba80 0x560f9a42de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 4 client.15603 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560f9a299540 con 0x560f9a42ba80 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a42ba80 0x560f9a42de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b02ff5640 10 client.15603 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b1e7fc640 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a42ba80 0x560f9a42de70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8403c1ea1590ebe server_cookie=546f5abd63a0d23 in_seq=0 out_seq=0 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b02ff5640 1 -- 172.21.15.78:0/1654271050 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560f9a299540 con 0x560f9a42ba80 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b02ff5640 10 client.15603 handle_command_reply: tid=0 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 client.15603 shutdown 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 msgr2=0x560f9a3be5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3be5d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8b100cd070 tx=0x7f8b100cca90 comp rx=0 tx=0).stop 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 shutdown_connections 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x560f9a42ba80 0x560f9a42de70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f9a3b4cd0 0x560f9a3bfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560f9a3afb60 0x560f9a3beb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b02ff5640 0 client.15603 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 --2- 172.21.15.78:0/1654271050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f9a3af170 0x560f9a3be5d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 >> 172.21.15.78:0/1654271050 conn(0x560f9a3aa6e0 msgr2=0x560f9a3ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 shutdown_connections 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 1 -- 172.21.15.78:0/1654271050 wait complete. 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.697+0000 7f8b27f67740 20 client.15603 trim_cache size 0 max 16384 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "0979cb6b-2d19-4622-8f9b-9566da68c747": { 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-16T11:06:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "0979cb6b-2d19-4622-8f9b-9566da68c747", 2024-03-16T11:06:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-16T11:06:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:06:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:06:32.705 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:06:32.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b080a2890 msgr2=0x7f8b080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b080a2890 0x7f8b080a4d50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009fd0 tx=0x7f8b18004a40 comp rx=0 tx=0).stop 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b20100700 msgr2=0x7f8b2006a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b20100700 0x7f8b2006a980 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000e9c0 tx=0x7f8b1000ee80 comp rx=0 tx=0).stop 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 shutdown_connections 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f8b080a2890 0x7f8b080a4d50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b201022b0 0x7f8b2019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b201018f0 0x7f8b2019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 --2- 172.21.15.78:0/2438037601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b20100700 0x7f8b2006a980 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 >> 172.21.15.78:0/2438037601 conn(0x7f8b200fbc90 msgr2=0x7f8b20104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 shutdown_connections 2024-03-16T11:06:32.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:06:32.698+0000 7f8b25377640 1 -- 172.21.15.78:0/2438037601 wait complete. 2024-03-16T11:06:32.709 INFO:tasks.cephfs.filesystem:scrub status for tag:0979cb6b-2d19-4622-8f9b-9566da68c747 - {'path': '/', 'tag': '0979cb6b-2d19-4622-8f9b-9566da68c747', 'options': 'recursive,force'} 2024-03-16T11:06:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:33 smithi078 ceph-mon[32235]: pgmap v365: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:33 smithi078 ceph-mon[32235]: scrub queued for path: / 2024-03-16T11:06:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:33 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:33 smithi078 ceph-mon[32235]: scrub summary: active paths [/] 2024-03-16T11:06:33.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:33 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:33.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:33 smithi098 ceph-mon[35848]: pgmap v365: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:33.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:33 smithi098 ceph-mon[35848]: scrub queued for path: / 2024-03-16T11:06:33.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:33 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:33.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:33 smithi098 ceph-mon[35848]: scrub summary: active paths [/] 2024-03-16T11:06:33.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:33 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:33 smithi153 ceph-mon[35521]: pgmap v365: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:33 smithi153 ceph-mon[35521]: scrub queued for path: / 2024-03-16T11:06:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:33 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:33 smithi153 ceph-mon[35521]: scrub summary: active paths [/] 2024-03-16T11:06:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:33 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:06:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:35 smithi098 ceph-mon[35848]: pgmap v366: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:35 smithi098 ceph-mon[35848]: scrub summary: idle 2024-03-16T11:06:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:35 smithi153 ceph-mon[35521]: pgmap v366: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:35 smithi153 ceph-mon[35521]: scrub summary: idle 2024-03-16T11:06:35.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:35 smithi078 ceph-mon[32235]: pgmap v366: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:35.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:35 smithi078 ceph-mon[32235]: scrub summary: idle 2024-03-16T11:06:37.361 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:37 smithi078 ceph-mon[32235]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T11:06:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:37 smithi098 ceph-mon[35848]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T11:06:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:37 smithi153 ceph-mon[35521]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T11:06:39.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:39 smithi098 ceph-mon[35848]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:39.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 -- 172.21.15.78:0/2388256762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7368100700 msgr2=0x7f7368100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 --2- 172.21.15.78:0/2388256762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7368100700 0x7f7368100b20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f735802bc30 comp rx=0 tx=0).stop 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 -- 172.21.15.78:0/2388256762 shutdown_connections 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 --2- 172.21.15.78:0/2388256762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73681022b0 0x7f7368108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 --2- 172.21.15.78:0/2388256762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73681018f0 0x7f7368101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 --2- 172.21.15.78:0/2388256762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7368100700 0x7f7368100b20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 -- 172.21.15.78:0/2388256762 >> 172.21.15.78:0/2388256762 conn(0x7f73680fbc70 msgr2=0x7f73680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 -- 172.21.15.78:0/2388256762 shutdown_connections 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.514+0000 7f736e9f4640 1 -- 172.21.15.78:0/2388256762 wait complete. 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 Processor -- start 2024-03-16T11:06:39.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 -- start start 2024-03-16T11:06:39.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7368100700 0x7f73681968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73681018f0 0x7f7368196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73681022b0 0x7f736819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f736810c230 con 0x7f73681022b0 2024-03-16T11:06:39.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f736810c0b0 con 0x7f73681018f0 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.515+0000 7f736e9f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f736810c3b0 con 0x7f7368100700 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73681022b0 0x7f736819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73681022b0 0x7f736819de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47712/0 (socket says 172.21.15.78:47712) 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 -- 172.21.15.78:0/4012918566 learned_addr learned my addr 172.21.15.78:0/4012918566 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f73677fe640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73681018f0 0x7f7368196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f7367fff640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7368100700 0x7f73681968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 -- 172.21.15.78:0/4012918566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7368100700 msgr2=0x7f73681968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7368100700 0x7f73681968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 -- 172.21.15.78:0/4012918566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73681018f0 msgr2=0x7f7368196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73681018f0 0x7f7368196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 -- 172.21.15.78:0/4012918566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736819e4d0 con 0x7f73681022b0 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f73677fe640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73681018f0 0x7f7368196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736cf91640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73681022b0 0x7f736819de60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f736000eb10 tx=0x7f736000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:39.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f73657fa640 1 -- 172.21.15.78:0/4012918566 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360004b20 con 0x7f73681022b0 2024-03-16T11:06:39.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736819e7c0 con 0x7f73681022b0 2024-03-16T11:06:39.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f73657fa640 1 -- 172.21.15.78:0/4012918566 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7360004cc0 con 0x7f73681022b0 2024-03-16T11:06:39.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736819ed60 con 0x7f73681022b0 2024-03-16T11:06:39.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.516+0000 7f73657fa640 1 -- 172.21.15.78:0/4012918566 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360010690 con 0x7f73681022b0 2024-03-16T11:06:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73657fa640 1 -- 172.21.15.78:0/4012918566 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f7360010830 con 0x7f73681022b0 2024-03-16T11:06:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 Processor -- start 2024-03-16T11:06:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 -- start start 2024-03-16T11:06:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7771340 0x562ba7771760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba7776830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7776ea0 0x562ba7779330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562ba76d6a40 con 0x562ba7776ea0 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ba767f920 con 0x562ba7771d30 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.519+0000 7f73715e7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562ba76d2ae0 con 0x562ba7771340 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f73677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7776ea0 0x562ba7779330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba7776830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba7776830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:46904/0 (socket says 172.21.15.78:46904) 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f73677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7776ea0 0x562ba7779330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47724/0 (socket says 172.21.15.78:47724) 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f73657fa640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f73500a2940 0x7f73500a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 -- 172.21.15.78:0/2424727299 learned_addr learned my addr 172.21.15.78:0/2424727299 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f73677fe640 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7771340 0x562ba7771760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f736cf91640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f73500a2940 0x7f73500a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 -- 172.21.15.78:0/2424727299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7771340 msgr2=0x562ba7771760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7771340 0x562ba7771760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 -- 172.21.15.78:0/2424727299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7776ea0 msgr2=0x562ba7779330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7776ea0 0x562ba7779330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 -- 172.21.15.78:0/2424727299 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ba7697460 con 0x562ba7771d30 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f73657fa640 1 -- 172.21.15.78:0/4012918566 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f73600cf330 con 0x7f73681022b0 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f7367fff640 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba7776830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f7358009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=284d5f983e491693 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f734e7fc640 1 -- 172.21.15.78:0/2424727299 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735802c8e0 con 0x562ba7771d30 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f736cf91640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f73500a2940 0x7f73500a4e00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f736004afd0 tx=0x7f736001cea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f734e7fc640 1 -- 172.21.15.78:0/2424727299 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f735802ca80 con 0x562ba7771d30 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.520+0000 7f734e7fc640 1 -- 172.21.15.78:0/2424727299 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73580046c0 con 0x562ba7771d30 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 -- 172.21.15.78:0/2424727299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 msgr2=0x562ba7776830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba7776830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f7358009560 comp rx=0 tx=0).stop 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 -- 172.21.15.78:0/2424727299 shutdown_connections 2024-03-16T11:06:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7776ea0 0x562ba7779330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba7776830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 --2- 172.21.15.78:0/2424727299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7771340 0x562ba7771760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 -- 172.21.15.78:0/2424727299 >> 172.21.15.78:0/2424727299 conn(0x562ba776c8b0 msgr2=0x562ba776ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 -- 172.21.15.78:0/2424727299 shutdown_connections 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 -- 172.21.15.78:0/2424727299 wait complete. 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.521+0000 7f73715e7740 1 -- start start 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 1 Processor -- start 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771340 0x562ba777b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771d30 0x562ba777bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba7782ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562ba7779df0 con 0x562ba7771d30 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ba7692d70 con 0x562ba7771340 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562ba7779f70 con 0x562ba7776ea0 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73715e7740 10 client.0 _ll_register_callbacks cb 0x562ba73eaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771d30 0x562ba777bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771d30 0x562ba777bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47732/0 (socket says 172.21.15.78:47732) 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 -- 172.21.15.78:0/1469585605 learned_addr learned my addr 172.21.15.78:0/1469585605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73677fe640 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba7782ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f73677fe640 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771340 0x562ba777b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 -- 172.21.15.78:0/1469585605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 msgr2=0x562ba7782ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba7782ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 -- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771340 msgr2=0x562ba777b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771340 0x562ba777b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 -- 172.21.15.78:0/1469585605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ba73eb210 con 0x562ba7771d30 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.522+0000 7f7367fff640 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771d30 0x562ba777bb60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f735802c170 tx=0x7f735802c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5a5495976801b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73477fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73477fe640 1 -- 172.21.15.78:0/1469585605 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358004450 con 0x562ba7771d30 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73477fe640 1 -- 172.21.15.78:0/1469585605 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f73580035f0 con 0x562ba7771d30 2024-03-16T11:06:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f735802ca80 con 0x562ba7771d30 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73477fe640 1 -- 172.21.15.78:0/1469585605 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f73580037f0 con 0x562ba7771d30 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 10 client.15615 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562ba74b7620 con 0x562ba7771d30 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73477fe640 1 -- 172.21.15.78:0/1469585605 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f735802ca80 con 0x562ba7771d30 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73477fe640 1 -- 172.21.15.78:0/1469585605 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7358004c20 con 0x562ba7771d30 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 10 client.15615 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 10 client.15615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 20 client.15615 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba7787090 0x562ba7789480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 4 client.15615 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ba76e3f80 con 0x562ba7787090 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.523+0000 7f73677fe640 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba7787090 0x562ba7789480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.524+0000 7f73677fe640 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba7787090 0x562ba7789480 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d2133aa090aa553 server_cookie=c826999a1bc5924f in_seq=0 out_seq=0 2024-03-16T11:06:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.524+0000 7f73477fe640 10 client.15615 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73477fe640 1 -- 172.21.15.78:0/1469585605 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562ba76e3f80 con 0x562ba7787090 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73477fe640 10 client.15615 handle_command_reply: tid=0 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 client.15615 shutdown 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771d30 msgr2=0x562ba777bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771d30 0x562ba777bb60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f735802c170 tx=0x7f735802c3b0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 shutdown_connections 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba7787090 0x562ba7789480 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba7782ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73477fe640 0 client.15615 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771d30 0x562ba777bb60 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 --2- 172.21.15.78:0/1469585605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771340 0x562ba777b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 >> 172.21.15.78:0/1469585605 conn(0x562ba776c8b0 msgr2=0x562ba776bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 shutdown_connections 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 1 -- 172.21.15.78:0/1469585605 wait complete. 2024-03-16T11:06:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.525+0000 7f73715e7740 20 client.15615 trim_cache size 0 max 16384 2024-03-16T11:06:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 Processor -- start 2024-03-16T11:06:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 -- start start 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba7770220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba778dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba778e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f73681a9270 con 0x562ba7771340 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73681a6b40 con 0x562ba7771d30 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73715e7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73681a4970 con 0x562ba7776ea0 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba7770220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba7770220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47748/0 (socket says 172.21.15.78:47748) 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 -- 172.21.15.78:0/2250556839 learned_addr learned my addr 172.21.15.78:0/2250556839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f7367fff640 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba778dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba778e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 -- 172.21.15.78:0/2250556839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 msgr2=0x562ba778e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba778e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 -- 172.21.15.78:0/2250556839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 msgr2=0x562ba778dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba778dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 -- 172.21.15.78:0/2250556839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ba7789ae0 con 0x562ba7771340 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.540+0000 7f73677fe640 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba7770220 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f735c0058c0 tx=0x7f735c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e15f23d2d6ab84 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.541+0000 7f734dffb640 1 -- 172.21.15.78:0/2250556839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c003a40 con 0x562ba7771340 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.541+0000 7f734dffb640 1 -- 172.21.15.78:0/2250556839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f735c02b020 con 0x562ba7771340 2024-03-16T11:06:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.541+0000 7f734dffb640 1 -- 172.21.15.78:0/2250556839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c023540 con 0x562ba7771340 2024-03-16T11:06:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:39 smithi153 ceph-mon[35521]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 -- 172.21.15.78:0/2250556839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 msgr2=0x562ba7770220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba7770220 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f735c0058c0 tx=0x7f735c0046c0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 -- 172.21.15.78:0/2250556839 shutdown_connections 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba778e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba778dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 --2- 172.21.15.78:0/2250556839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba7770220 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 -- 172.21.15.78:0/2250556839 >> 172.21.15.78:0/2250556839 conn(0x562ba776c8b0 msgr2=0x562ba776fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 -- 172.21.15.78:0/2250556839 shutdown_connections 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 -- 172.21.15.78:0/2250556839 wait complete. 2024-03-16T11:06:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 -- start start 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.542+0000 7f73715e7740 1 Processor -- start 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba77e9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba77e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba77f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f73681a5000 con 0x562ba7771340 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73681a6470 con 0x562ba7771d30 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73681a56d0 con 0x562ba7776ea0 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 10 client.0 _ll_register_callbacks cb 0x7f735c009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba77e9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba77e9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47754/0 (socket says 172.21.15.78:47754) 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 -- 172.21.15.78:0/1873370269 learned_addr learned my addr 172.21.15.78:0/1873370269 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f7367fff640 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba77e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba77f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 -- 172.21.15.78:0/1873370269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 msgr2=0x562ba77f0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba77f0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 -- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 msgr2=0x562ba77e9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba77e9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 -- 172.21.15.78:0/1873370269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ba77f10b0 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73677fe640 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba77e9540 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f735c02a040 tx=0x7f735c005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5fa0802cc8f616 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73477fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73477fe640 1 -- 172.21.15.78:0/1873370269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c026040 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73477fe640 1 -- 172.21.15.78:0/1873370269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f735c0261e0 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f735c02b020 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73477fe640 1 -- 172.21.15.78:0/1873370269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f735c0263e0 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 10 client.15621 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.543+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562ba77f23c0 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73477fe640 1 -- 172.21.15.78:0/1873370269 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f735c02b020 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73477fe640 1 -- 172.21.15.78:0/1873370269 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f735c039070 con 0x562ba7771340 2024-03-16T11:06:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73715e7740 10 client.15621 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73715e7740 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73715e7740 20 client.15621 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73715e7740 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba77f2d70 0x562ba77f5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73715e7740 4 client.15621 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562ba76de2d0 con 0x562ba77f2d70 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f7367fff640 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba77f2d70 0x562ba77f5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f73477fe640 10 client.15621 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.544+0000 7f7367fff640 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba77f2d70 0x562ba77f5190 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce91257bfc26ae38 server_cookie=9fd434714a2faccc in_seq=0 out_seq=0 2024-03-16T11:06:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73477fe640 1 -- 172.21.15.78:0/1873370269 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562ba76de2d0 con 0x562ba77f2d70 2024-03-16T11:06:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73477fe640 10 client.15621 handle_command_reply: tid=0 2024-03-16T11:06:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 client.15621 shutdown 2024-03-16T11:06:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 msgr2=0x562ba77e9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba77e9540 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f735c02a040 tx=0x7f735c005810 comp rx=0 tx=0).stop 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 shutdown_connections 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x562ba77f2d70 0x562ba77f5190 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73477fe640 0 client.15621 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562ba7776ea0 0x562ba77f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ba7771d30 0x562ba77e9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 --2- 172.21.15.78:0/1873370269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ba7771340 0x562ba77e9540 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 >> 172.21.15.78:0/1873370269 conn(0x562ba776c8b0 msgr2=0x562ba776fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.547+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 shutdown_connections 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f73715e7740 1 -- 172.21.15.78:0/1873370269 wait complete. 2024-03-16T11:06:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f73715e7740 20 client.15621 trim_cache size 0 max 16384 2024-03-16T11:06:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:06:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:06:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:06:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:06:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f73500a2940 msgr2=0x7f73500a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f73500a2940 0x7f73500a4e00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f736004afd0 tx=0x7f736001cea0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73681022b0 msgr2=0x7f736819de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73681022b0 0x7f736819de60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f736000eb10 tx=0x7f736000c2d0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 shutdown_connections 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73681022b0 0x7f736819de60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73681018f0 0x7f7368196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f73500a2940 0x7f73500a4e00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 --2- 172.21.15.78:0/4012918566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7368100700 0x7f73681968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 >> 172.21.15.78:0/4012918566 conn(0x7f73680fbc70 msgr2=0x7f7368104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 shutdown_connections 2024-03-16T11:06:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:39.548+0000 7f736e9f4640 1 -- 172.21.15.78:0/4012918566 wait complete. 2024-03-16T11:06:39.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:39 smithi078 ceph-mon[32235]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:41 smithi098 ceph-mon[35848]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:41 smithi153 ceph-mon[35521]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:41.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:41 smithi078 ceph-mon[32235]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:43.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:43 smithi153 ceph-mon[35521]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-16T11:06:43.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:43 smithi098 ceph-mon[35848]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-16T11:06:43.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:43 smithi078 ceph-mon[32235]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-16T11:06:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:45 smithi098 ceph-mon[35848]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:45 smithi153 ceph-mon[35521]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:45.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:45 smithi078 ceph-mon[32235]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:47 smithi098 ceph-mon[35848]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T11:06:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:47 smithi153 ceph-mon[35521]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T11:06:47.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:47 smithi078 ceph-mon[32235]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T11:06:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:49 smithi098 ceph-mon[35848]: pgmap v373: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:06:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:49 smithi153 ceph-mon[35521]: pgmap v373: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:06:49.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:49 smithi078 ceph-mon[32235]: pgmap v373: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:06:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:51 smithi098 ceph-mon[35848]: pgmap v374: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:51 smithi153 ceph-mon[35521]: pgmap v374: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:51.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:51 smithi078 ceph-mon[32235]: pgmap v374: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:06:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 -- 172.21.15.78:0/3722171604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 msgr2=0x7ff7b4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3722171604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4101d70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009860 tx=0x7ff7a802bb70 comp rx=0 tx=0).stop 2024-03-16T11:06:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 -- 172.21.15.78:0/3722171604 shutdown_connections 2024-03-16T11:06:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3722171604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7b41022b0 0x7ff7b4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3722171604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4101d70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3722171604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7b4100700 0x7ff7b4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 -- 172.21.15.78:0/3722171604 >> 172.21.15.78:0/3722171604 conn(0x7ff7b40fbc70 msgr2=0x7ff7b40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 -- 172.21.15.78:0/3722171604 shutdown_connections 2024-03-16T11:06:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 -- 172.21.15.78:0/3722171604 wait complete. 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.640+0000 7ff7b916f640 1 Processor -- start 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b916f640 1 -- start start 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b916f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7b4100700 0x7ff7b4197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b916f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b916f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7b41022b0 0x7ff7b419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b916f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b410c230 con 0x7ff7b41022b0 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b916f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b410c0b0 con 0x7ff7b41018f0 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b916f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b410c3b0 con 0x7ff7b4100700 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7b4100700 0x7ff7b4197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7b41022b0 0x7ff7b419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43484/0 (socket says 172.21.15.78:43484) 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 -- 172.21.15.78:0/3948809397 learned_addr learned my addr 172.21.15.78:0/3948809397 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7b4100700 0x7ff7b4197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:44824/0 (socket says 172.21.15.78:44824) 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 -- 172.21.15.78:0/3948809397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7b4100700 msgr2=0x7ff7b4197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7b4100700 0x7ff7b4197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 -- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7b41022b0 msgr2=0x7ff7b419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7b41022b0 0x7ff7b419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b259c640 1 -- 172.21.15.78:0/3948809397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b419f5a0 con 0x7ff7b41018f0 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.641+0000 7ff7b359e640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7b41022b0 0x7ff7b419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.642+0000 7ff7b259c640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4197ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8005940 tx=0x7ff7a8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.642+0000 7ff79bfff640 1 -- 172.21.15.78:0/3948809397 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8041020 con 0x7ff7b41018f0 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.642+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7b4102c60 con 0x7ff7b41018f0 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.642+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7b41031a0 con 0x7ff7b41018f0 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.642+0000 7ff79bfff640 1 -- 172.21.15.78:0/3948809397 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7a802c9c0 con 0x7ff7b41018f0 2024-03-16T11:06:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.642+0000 7ff79bfff640 1 -- 172.21.15.78:0/3948809397 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8039040 con 0x7ff7b41018f0 2024-03-16T11:06:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.645+0000 7ff79bfff640 1 -- 172.21.15.78:0/3948809397 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7ff7a8044020 con 0x7ff7b41018f0 2024-03-16T11:06:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.645+0000 7ff7bbe31740 1 Processor -- start 2024-03-16T11:06:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.645+0000 7ff7bbe31740 1 -- start start 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.645+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e58d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.645+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e592830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e595330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558d4e4f2a40 con 0x558d4e58d340 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558d4e49b920 con 0x558d4e592ea0 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558d4e4eeae0 con 0x558d4e58dd30 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e58d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff79bfff640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff79c0a2940 0x7ff79c0a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e58d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39052/0 (socket says 172.21.15.78:39052) 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 -- 172.21.15.78:0/2464569568 learned_addr learned my addr 172.21.15.78:0/2464569568 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b259c640 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e595330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e592830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff79c0a2940 0x7ff79c0a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff79bfff640 1 -- 172.21.15.78:0/3948809397 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff7a80f0410 con 0x7ff7b41018f0 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 -- 172.21.15.78:0/2464569568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 msgr2=0x558d4e592830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e592830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 -- 172.21.15.78:0/2464569568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 msgr2=0x558d4e595330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e595330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 -- 172.21.15.78:0/2464569568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d4e4b3460 con 0x558d4e58d340 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e592830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:06:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.646+0000 7ff7b359e640 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e58d760 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009fd0 tx=0x7ff7ac00b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ff6ba58fee3a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7997fa640 1 -- 172.21.15.78:0/2464569568 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac023030 con 0x558d4e58d340 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7b359e640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff79c0a2940 0x7ff79c0a4e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac027000 tx=0x7ff7ac024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7997fa640 1 -- 172.21.15.78:0/2464569568 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7ac010b20 con 0x558d4e58d340 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7997fa640 1 -- 172.21.15.78:0/2464569568 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac01fab0 con 0x558d4e58d340 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 -- 172.21.15.78:0/2464569568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 msgr2=0x558d4e58d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e58d760 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009fd0 tx=0x7ff7ac00b750 comp rx=0 tx=0).stop 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 -- 172.21.15.78:0/2464569568 shutdown_connections 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e595330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e592830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/2464569568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e58d760 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 -- 172.21.15.78:0/2464569568 >> 172.21.15.78:0/2464569568 conn(0x558d4e5888b0 msgr2=0x558d4e588cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 -- 172.21.15.78:0/2464569568 shutdown_connections 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.647+0000 7ff7bbe31740 1 -- 172.21.15.78:0/2464569568 wait complete. 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 -- start start 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 Processor -- start 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e59fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e5a0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e5a7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558d4e595e50 con 0x558d4e58d340 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558d4e4aed70 con 0x558d4e592ea0 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558d4e595fd0 con 0x558d4e58dd30 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7bbe31740 10 client.0 _ll_register_callbacks cb 0x558d4e206a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e59fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e59fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39058/0 (socket says 172.21.15.78:39058) 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b2d9d640 1 -- 172.21.15.78:0/4127977560 learned_addr learned my addr 172.21.15.78:0/4127977560 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b359e640 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e5a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b259c640 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e5a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b2d9d640 1 -- 172.21.15.78:0/4127977560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 msgr2=0x558d4e5a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e5a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b2d9d640 1 -- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 msgr2=0x558d4e5a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.648+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e5a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff7b2d9d640 1 -- 172.21.15.78:0/4127977560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d4e207210 con 0x558d4e58d340 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e59fee0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4005fd0 tx=0x7ff7a4011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99138c15407e5e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff796ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff796ff5640 1 -- 172.21.15.78:0/4127977560 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a401e020 con 0x558d4e58d340 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff796ff5640 1 -- 172.21.15.78:0/4127977560 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff7a400fb90 con 0x558d4e58d340 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7ac010b20 con 0x558d4e58d340 2024-03-16T11:06:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff796ff5640 1 -- 172.21.15.78:0/4127977560 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff7a400fd90 con 0x558d4e58d340 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff7bbe31740 10 client.15633 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558d4e2d3620 con 0x558d4e58d340 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff796ff5640 1 -- 172.21.15.78:0/4127977560 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7ac010b20 con 0x558d4e58d340 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.649+0000 7ff796ff5640 1 -- 172.21.15.78:0/4127977560 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff7a4019070 con 0x558d4e58d340 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7bbe31740 10 client.15633 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7bbe31740 10 client.15633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7bbe31740 20 client.15633 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e5ac220 0x558d4e5ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7bbe31740 4 client.15633 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d4e4fff80 con 0x558d4e5ac220 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7b259c640 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e5ac220 0x558d4e5ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff796ff5640 10 client.15633 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.650+0000 7ff7b259c640 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e5ac220 0x558d4e5ae6c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab3d4fbbdf88fd2f server_cookie=268693bb80805184 in_seq=0 out_seq=0 2024-03-16T11:06:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff796ff5640 1 -- 172.21.15.78:0/4127977560 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558d4e4fff80 con 0x558d4e5ac220 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff796ff5640 10 client.15633 handle_command_reply: tid=0 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 client.15633 shutdown 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 msgr2=0x558d4e59fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e59fee0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4005fd0 tx=0x7ff7a4011b30 comp rx=0 tx=0).stop 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 shutdown_connections 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e5ac220 0x558d4e5ae6c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e592ea0 0x558d4e5a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58dd30 0x558d4e5a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4127977560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e58d340 0x558d4e59fee0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 >> 172.21.15.78:0/4127977560 conn(0x558d4e5888b0 msgr2=0x558d4e587bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff796ff5640 0 client.15633 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 shutdown_connections 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4127977560 wait complete. 2024-03-16T11:06:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.651+0000 7ff7bbe31740 20 client.15633 trim_cache size 0 max 16384 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 Processor -- start 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 -- start start 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e596370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b41a9310 con 0x558d4e592ea0 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b41a6be0 con 0x558d4e58dd30 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b41a4a10 con 0x558d4e58d340 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.671+0000 7ff7b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43502/0 (socket says 172.21.15.78:43502) 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 -- 172.21.15.78:0/414087521 learned_addr learned my addr 172.21.15.78:0/414087521 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b359e640 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e596370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 -- 172.21.15.78:0/414087521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 msgr2=0x558d4e5a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 -- 172.21.15.78:0/414087521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 msgr2=0x558d4e596370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e596370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 -- 172.21.15.78:0/414087521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d4e57a140 con 0x558d4e58dd30 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b359e640 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e596370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff7b259c640 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a7420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8003fd0 tx=0x7ff7a808fa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ce62fba252a67c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff798ff9640 1 -- 172.21.15.78:0/414087521 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80bd460 con 0x558d4e58dd30 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff798ff9640 1 -- 172.21.15.78:0/414087521 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7a80bcd60 con 0x558d4e58dd30 2024-03-16T11:06:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.672+0000 7ff798ff9640 1 -- 172.21.15.78:0/414087521 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8003760 con 0x558d4e58dd30 2024-03-16T11:06:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 -- 172.21.15.78:0/414087521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 msgr2=0x558d4e5a7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a7420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8003fd0 tx=0x7ff7a808fa90 comp rx=0 tx=0).stop 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 -- 172.21.15.78:0/414087521 shutdown_connections 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e596370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a7420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/414087521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 -- 172.21.15.78:0/414087521 >> 172.21.15.78:0/414087521 conn(0x558d4e5888b0 msgr2=0x558d4e59fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 -- 172.21.15.78:0/414087521 shutdown_connections 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 -- 172.21.15.78:0/414087521 wait complete. 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.673+0000 7ff7bbe31740 1 -- start start 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 1 Processor -- start 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e6105f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b41a50a0 con 0x558d4e592ea0 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b41a6510 con 0x558d4e58dd30 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b41a5770 con 0x558d4e58d340 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7bbe31740 10 client.0 _ll_register_callbacks cb 0x7ff7a4009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e6105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43510/0 (socket says 172.21.15.78:43510) 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 -- 172.21.15.78:0/4223117367 learned_addr learned my addr 172.21.15.78:0/4223117367 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 -- 172.21.15.78:0/4223117367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 msgr2=0x558d4e5a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 -- 172.21.15.78:0/4223117367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 msgr2=0x558d4e6105f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:06:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e6105f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 -- 172.21.15.78:0/4223117367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d4e5a7510 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff7b259c640 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a4440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7a80094d0 tx=0x7ff7a8005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3fe4bf428639cf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.674+0000 7ff796ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff796ff5640 1 -- 172.21.15.78:0/4223117367 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80c5500 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7a80bcd60 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff796ff5640 1 -- 172.21.15.78:0/4223117367 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff7a8003760 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff796ff5640 1 -- 172.21.15.78:0/4223117367 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff7a80bd460 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 10 client.25475 fetch_fsmap learned FSMap version 22 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558d4e611a10 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff796ff5640 1 -- 172.21.15.78:0/4223117367 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7a80bcd60 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff796ff5640 1 -- 172.21.15.78:0/4223117367 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff7a80a7040 con 0x558d4e58dd30 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 10 client.25475 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 10 client.25475 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 20 client.25475 populate_metadata read hostname 'smithi078' 2024-03-16T11:06:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e612360 0x558d4e614780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:06:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 4 client.25475 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:06:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d4e4fa2d0 con 0x558d4e612360 2024-03-16T11:06:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.675+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e612360 0x558d4e614780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:06:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.676+0000 7ff796ff5640 10 client.25475 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.676+0000 7ff7b2d9d640 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e612360 0x558d4e614780 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ee433cbf47a275f server_cookie=7c3197ffc71fb7a3 in_seq=0 out_seq=0 2024-03-16T11:06:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.678+0000 7ff796ff5640 1 -- 172.21.15.78:0/4223117367 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d4e4fa2d0 con 0x558d4e612360 2024-03-16T11:06:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.678+0000 7ff796ff5640 10 client.25475 handle_command_reply: tid=0 2024-03-16T11:06:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 client.25475 shutdown 2024-03-16T11:06:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 msgr2=0x558d4e5a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a4440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7a80094d0 tx=0x7ff7a8005750 comp rx=0 tx=0).stop 2024-03-16T11:06:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 shutdown_connections 2024-03-16T11:06:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x558d4e612360 0x558d4e614780 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d4e592ea0 0x558d4e6105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558d4e58dd30 0x558d4e5a4440 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff796ff5640 0 client.25475 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 --2- 172.21.15.78:0/4223117367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d4e58d340 0x558d4e5a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 >> 172.21.15.78:0/4223117367 conn(0x558d4e5888b0 msgr2=0x558d4e58bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 shutdown_connections 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 1 -- 172.21.15.78:0/4223117367 wait complete. 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7bbe31740 20 client.25475 trim_cache size 0 max 16384 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:06:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:06:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:06:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff79c0a2940 msgr2=0x7ff79c0a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff79c0a2940 0x7ff79c0a4e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac027000 tx=0x7ff7ac024000 comp rx=0 tx=0).stop 2024-03-16T11:06:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 msgr2=0x7ff7b4197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.679+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4197ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8005940 tx=0x7ff7a8004270 comp rx=0 tx=0).stop 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 shutdown_connections 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7b41022b0 0x7ff7b419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b41018f0 0x7ff7b4197ec0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff79c0a2940 0x7ff79c0a4e00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 --2- 172.21.15.78:0/3948809397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7b4100700 0x7ff7b4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 >> 172.21.15.78:0/3948809397 conn(0x7ff7b40fbc70 msgr2=0x7ff7b4068e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 shutdown_connections 2024-03-16T11:06:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:06:52.680+0000 7ff7b916f640 1 -- 172.21.15.78:0/3948809397 wait complete. 2024-03-16T11:06:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:53 smithi098 ceph-mon[35848]: pgmap v375: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:53 smithi153 ceph-mon[35521]: pgmap v375: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:53.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:53 smithi078 ceph-mon[32235]: pgmap v375: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:55 smithi098 ceph-mon[35848]: pgmap v376: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:06:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:55 smithi153 ceph-mon[35521]: pgmap v376: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:06:55.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:55 smithi078 ceph-mon[32235]: pgmap v376: 97 pgs: 97 active+clean; 661 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:06:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:57 smithi098 ceph-mon[35848]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:57 smithi153 ceph-mon[35521]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:57.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:57 smithi078 ceph-mon[32235]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:06:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:06:59 smithi098 ceph-mon[35848]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:06:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:06:59 smithi153 ceph-mon[35521]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:06:59.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:06:59 smithi078 ceph-mon[32235]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:07:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:01 smithi098 ceph-mon[35848]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:07:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:01 smithi153 ceph-mon[35521]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:07:01.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:01 smithi078 ceph-mon[32235]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:07:02.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:07:02.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:07:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.822+0000 7f4a0d668640 1 -- 172.21.15.78:0/301485883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081018d0 msgr2=0x7f4a08101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.822+0000 7f4a0d668640 1 --2- 172.21.15.78:0/301485883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081018d0 0x7f4a08101d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009900 tx=0x7f49fc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 -- 172.21.15.78:0/301485883 shutdown_connections 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 --2- 172.21.15.78:0/301485883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a08102290 0x7f4a08108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 --2- 172.21.15.78:0/301485883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081018d0 0x7f4a08101d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 --2- 172.21.15.78:0/301485883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081006e0 0x7f4a08100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 -- 172.21.15.78:0/301485883 >> 172.21.15.78:0/301485883 conn(0x7f4a080fbc70 msgr2=0x7f4a080fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 -- 172.21.15.78:0/301485883 shutdown_connections 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 -- 172.21.15.78:0/301485883 wait complete. 2024-03-16T11:07:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.823+0000 7f4a0d668640 1 Processor -- start 2024-03-16T11:07:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a0d668640 1 -- start start 2024-03-16T11:07:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a0d668640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081006e0 0x7f4a08197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a0d668640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081018d0 0x7f4a08197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a0d668640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a08102290 0x7f4a0819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081018d0 0x7f4a08197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081006e0 0x7f4a08197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081018d0 0x7f4a08197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55026/0 (socket says 172.21.15.78:55026) 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 -- 172.21.15.78:0/441127229 learned_addr learned my addr 172.21.15.78:0/441127229 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0810c210 con 0x7f4a081018d0 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0810c090 con 0x7f4a081006e0 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0810c390 con 0x7f4a08102290 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081006e0 0x7f4a08197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43008/0 (socket says 172.21.15.78:43008) 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a077fe640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a08102290 0x7f4a0819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 -- 172.21.15.78:0/441127229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a08102290 msgr2=0x7f4a0819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a08102290 0x7f4a0819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 -- 172.21.15.78:0/441127229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081006e0 msgr2=0x7f4a08197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081006e0 0x7f4a08197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a067fc640 1 -- 172.21.15.78:0/441127229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0819f650 con 0x7f4a081018d0 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.824+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081006e0 0x7f4a08197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.825+0000 7f4a067fc640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081018d0 0x7f4a08197f70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f49fc006010 tx=0x7f49fc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.825+0000 7f49effff640 1 -- 172.21.15.78:0/441127229 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc043020 con 0x7f4a081018d0 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.825+0000 7f49effff640 1 -- 172.21.15.78:0/441127229 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49fc0043d0 con 0x7f4a081018d0 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.825+0000 7f49effff640 1 -- 172.21.15.78:0/441127229 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc003b80 con 0x7f4a081018d0 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.826+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0819f8e0 con 0x7f4a081018d0 2024-03-16T11:07:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.826+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0819fde0 con 0x7f4a081018d0 2024-03-16T11:07:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.828+0000 7f4a1024e740 1 Processor -- start 2024-03-16T11:07:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.828+0000 7f4a1024e740 1 -- start start 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.828+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39464590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464b60 0x55ce39469660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 0x55ce3946c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a1024e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ce393c9500 con 0x55ce39464170 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a1024e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ce39371be0 con 0x55ce39469cd0 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a1024e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ce393c55a0 con 0x55ce39464b60 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39464590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39464590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55040/0 (socket says 172.21.15.78:55040) 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a077fe640 1 -- 172.21.15.78:0/1360363748 learned_addr learned my addr 172.21.15.78:0/1360363748 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464b60 0x55ce39469660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f49effff640 1 -- 172.21.15.78:0/441127229 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f49fc046020 con 0x7f4a081018d0 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a06ffd640 1 -- 172.21.15.78:0/1360363748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 msgr2=0x55ce3946c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 0x55ce3946c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a06ffd640 1 -- 172.21.15.78:0/1360363748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 msgr2=0x55ce39464590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39464590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a06ffd640 1 -- 172.21.15.78:0/1360363748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce393854b0 con 0x55ce39464b60 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a077fe640 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39464590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.829+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464b60 0x55ce39469660 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009fa0 tx=0x7f49f80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d66109e1a3db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.830+0000 7f49effff640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f49f00a2910 0x7f49f00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.830+0000 7f49ed7fa640 1 -- 172.21.15.78:0/1360363748 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8004310 con 0x55ce39464b60 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.830+0000 7f49ed7fa640 1 -- 172.21.15.78:0/1360363748 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49f80044b0 con 0x55ce39464b60 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.830+0000 7f49ed7fa640 1 -- 172.21.15.78:0/1360363748 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f80035f0 con 0x55ce39464b60 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.830+0000 7f4a077fe640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f49f00a2910 0x7f49f00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.830+0000 7f49effff640 1 -- 172.21.15.78:0/441127229 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f49fc0f0c80 con 0x7f4a081018d0 2024-03-16T11:07:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 -- 172.21.15.78:0/1360363748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464b60 msgr2=0x55ce39469660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a077fe640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f49f00a2910 0x7f49f00a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55ce394653c0 tx=0x7f4a00009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464b60 0x55ce39469660 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009fa0 tx=0x7f49f80118d0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 -- 172.21.15.78:0/1360363748 shutdown_connections 2024-03-16T11:07:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 0x55ce3946c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464b60 0x55ce39469660 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 --2- 172.21.15.78:0/1360363748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39464590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 -- 172.21.15.78:0/1360363748 >> 172.21.15.78:0/1360363748 conn(0x55ce3945f6e0 msgr2=0x55ce3945fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 -- 172.21.15.78:0/1360363748 shutdown_connections 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 -- 172.21.15.78:0/1360363748 wait complete. 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 -- start start 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.831+0000 7f4a1024e740 1 Processor -- start 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3946e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39464b60 0x55ce3946ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39469cd0 0x55ce39475cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a1024e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ce3946cc80 con 0x55ce39469cd0 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a1024e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ce39385160 con 0x55ce39464b60 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a06ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3946e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a1024e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ce3946ce00 con 0x55ce39464170 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39464b60 0x55ce3946ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a06ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3946e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54048/0 (socket says 172.21.15.78:54048) 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a06ffd640 1 -- 172.21.15.78:0/2860354549 learned_addr learned my addr 172.21.15.78:0/2860354549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a077fe640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39469cd0 0x55ce39475cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a1024e740 10 client.0 _ll_register_callbacks cb 0x55ce39369150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a06ffd640 1 -- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39464b60 msgr2=0x55ce3946ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39464b60 0x55ce3946ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.832+0000 7f4a06ffd640 1 -- 172.21.15.78:0/2860354549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39469cd0 msgr2=0x55ce39475cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39469cd0 0x55ce39475cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a06ffd640 1 -- 172.21.15.78:0/2860354549 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce391a77d0 con 0x55ce39464170 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f80044b0 con 0x55ce39464170 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a077fe640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39469cd0 0x55ce39475cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a067fc640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39464b60 0x55ce3946ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3946e740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f49f801a000 tx=0x7f49f80043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21dfa88f3de88c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f49eaff5640 10 client.25471 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f49eaff5640 1 -- 172.21.15.78:0/2860354549 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8032050 con 0x55ce39464170 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f49eaff5640 1 -- 172.21.15.78:0/2860354549 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f49f8003770 con 0x55ce39464170 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f49eaff5640 1 -- 172.21.15.78:0/2860354549 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f49f8027030 con 0x55ce39464170 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a1024e740 10 client.25471 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.833+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55ce390dd3f0 con 0x55ce39464170 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f49eaff5640 1 -- 172.21.15.78:0/2860354549 <== mon.1 v2:172.21.15.153:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f49f80039b0 con 0x55ce39464170 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a1024e740 10 client.25471 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a1024e740 10 client.25471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a1024e740 20 client.25471 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce3947a3e0 0x55ce3947c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a1024e740 4 client.25471 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce393d6a40 con 0x55ce3947a3e0 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a067fc640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce3947a3e0 0x55ce3947c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f49eaff5640 10 client.25471 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.834+0000 7f4a067fc640 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce3947a3e0 0x55ce3947c7f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72b141da5bf1968d server_cookie=89327f177fa41b91 in_seq=0 out_seq=0 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f49eaff5640 1 -- 172.21.15.78:0/2860354549 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ce393d6a40 con 0x55ce3947a3e0 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f49eaff5640 10 client.25471 handle_command_reply: tid=0 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 client.25471 shutdown 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 msgr2=0x55ce3946e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3946e740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f49f801a000 tx=0x7f49f80043b0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 shutdown_connections 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce3947a3e0 0x55ce3947c7f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39469cd0 0x55ce39475cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f49eaff5640 0 client.25471 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39464b60 0x55ce3946ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2860354549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3946e740 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 >> 172.21.15.78:0/2860354549 conn(0x55ce3945f6e0 msgr2=0x55ce3945e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:02.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 shutdown_connections 2024-03-16T11:07:02.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 1 -- 172.21.15.78:0/2860354549 wait complete. 2024-03-16T11:07:02.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.835+0000 7f4a1024e740 20 client.25471 trim_cache size 0 max 16384 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.869+0000 7f4a1024e740 1 Processor -- start 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.869+0000 7f4a1024e740 1 -- start start 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3945a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464b60 0x55ce39475c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 0x55ce3945fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a1024e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a081ab590 con 0x55ce39464b60 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a1024e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a081a8e60 con 0x55ce39469cd0 2024-03-16T11:07:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a1024e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a081a6c90 con 0x55ce39464170 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464b60 0x55ce39475c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a06ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3945a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464b60 0x55ce39475c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55056/0 (socket says 172.21.15.78:55056) 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 -- 172.21.15.78:0/2743443467 learned_addr learned my addr 172.21.15.78:0/2743443467 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a077fe640 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 0x55ce3945fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 -- 172.21.15.78:0/2743443467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 msgr2=0x55ce3945a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3945a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 -- 172.21.15.78:0/2743443467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 msgr2=0x55ce3945fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 0x55ce3945fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 -- 172.21.15.78:0/2743443467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce3909d0f0 con 0x55ce39464b60 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.870+0000 7f4a067fc640 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464b60 0x55ce39475c80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0b9f60 tx=0x7f49fc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdc9167b62a9f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.871+0000 7f49ecff9640 1 -- 172.21.15.78:0/2743443467 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0d5020 con 0x55ce39464b60 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.871+0000 7f49ecff9640 1 -- 172.21.15.78:0/2743443467 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49fc0bdc90 con 0x55ce39464b60 2024-03-16T11:07:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.871+0000 7f49ecff9640 1 -- 172.21.15.78:0/2743443467 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0bb920 con 0x55ce39464b60 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- 172.21.15.78:0/2743443467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464b60 msgr2=0x55ce39475c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464b60 0x55ce39475c80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0b9f60 tx=0x7f49fc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- 172.21.15.78:0/2743443467 shutdown_connections 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39469cd0 0x55ce3945fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464b60 0x55ce39475c80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 --2- 172.21.15.78:0/2743443467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39464170 0x55ce3945a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- 172.21.15.78:0/2743443467 >> 172.21.15.78:0/2743443467 conn(0x55ce3945f6e0 msgr2=0x55ce394631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- 172.21.15.78:0/2743443467 shutdown_connections 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- 172.21.15.78:0/2743443467 wait complete. 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- start start 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 Processor -- start 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39475770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39469cd0 0x55ce39475cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39472280 0x55ce39472700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a081a7320 con 0x55ce39464170 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a081a8790 con 0x55ce39472280 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.872+0000 7f4a1024e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a081a79f0 con 0x55ce39469cd0 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a1024e740 10 client.0 _ll_register_callbacks cb 0x7f4a00009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a06ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39475770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39472280 0x55ce39472700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a06ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39475770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55058/0 (socket says 172.21.15.78:55058) 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a06ffd640 1 -- 172.21.15.78:0/537602749 learned_addr learned my addr 172.21.15.78:0/537602749 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39472280 0x55ce39472700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43038/0 (socket says 172.21.15.78:43038) 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a067fc640 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39469cd0 0x55ce39475cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 -- 172.21.15.78:0/537602749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39469cd0 msgr2=0x55ce39475cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39469cd0 0x55ce39475cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 -- 172.21.15.78:0/537602749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 msgr2=0x55ce39475770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39475770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 -- 172.21.15.78:0/537602749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce393909d0 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39475770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f4a077fe640 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39472280 0x55ce39472700 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005fd0 tx=0x7f4a00004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bf1c38ccaf512ab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f49eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.873+0000 7f49eaff5640 1 -- 172.21.15.78:0/537602749 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a000043d0 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.874+0000 7f49eaff5640 1 -- 172.21.15.78:0/537602749 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f4a00004570 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.874+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49fc0bdc90 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.874+0000 7f49eaff5640 1 -- 172.21.15.78:0/537602749 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4a00004770 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.874+0000 7f4a1024e740 10 client.25490 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.874+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ce394e7fa0 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.874+0000 7f49eaff5640 1 -- 172.21.15.78:0/537602749 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49fc0bdc90 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.874+0000 7f49eaff5640 1 -- 172.21.15.78:0/537602749 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4a0001c480 con 0x55ce39472280 2024-03-16T11:07:02.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a1024e740 10 client.25490 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a1024e740 10 client.25490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a1024e740 20 client.25490 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a1024e740 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce394e8980 0x55ce394eada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a1024e740 4 client.25490 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ce393d0d90 con 0x55ce394e8980 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce394e8980 0x55ce394eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f49eaff5640 10 client.25490 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.875+0000 7f4a06ffd640 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce394e8980 0x55ce394eada0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38a027bed2a2f798 server_cookie=9d2ba0a941d59a42 in_seq=0 out_seq=0 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f49eaff5640 1 -- 172.21.15.78:0/537602749 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ce393d0d90 con 0x55ce394e8980 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f49eaff5640 10 client.25490 handle_command_reply: tid=0 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 client.25490 shutdown 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39472280 msgr2=0x55ce39472700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39472280 0x55ce39472700 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005fd0 tx=0x7f4a00004ab0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 shutdown_connections 2024-03-16T11:07:02.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55ce394e8980 0x55ce394eada0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ce39472280 0x55ce39472700 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce39469cd0 0x55ce39475cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 --2- 172.21.15.78:0/537602749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce39464170 0x55ce39475770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 >> 172.21.15.78:0/537602749 conn(0x55ce3945f6e0 msgr2=0x55ce394631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f49eaff5640 0 client.25490 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 shutdown_connections 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 1 -- 172.21.15.78:0/537602749 wait complete. 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.876+0000 7f4a1024e740 20 client.25490 trim_cache size 0 max 16384 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-16T11:07:02.881 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-16T11:07:02.882 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:07:02.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f49f00a2910 msgr2=0x7f49f00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f49f00a2910 0x7f49f00a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55ce394653c0 tx=0x7f4a00009500 comp rx=0 tx=0).stop 2024-03-16T11:07:02.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081018d0 msgr2=0x7f4a08197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:02.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081018d0 0x7f4a08197f70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f49fc006010 tx=0x7f49fc005e40 comp rx=0 tx=0).stop 2024-03-16T11:07:02.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 shutdown_connections 2024-03-16T11:07:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f49f00a2910 0x7f49f00a4dd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a08102290 0x7f4a0819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a081018d0 0x7f4a08197f70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 --2- 172.21.15.78:0/441127229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a081006e0 0x7f4a08197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 >> 172.21.15.78:0/441127229 conn(0x7f4a080fbc70 msgr2=0x7f4a080fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 shutdown_connections 2024-03-16T11:07:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:02.877+0000 7f4a0d668640 1 -- 172.21.15.78:0/441127229 wait complete. 2024-03-16T11:07:02.892 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T11:07:02.892 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:07:03.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 -- 172.21.15.78:0/4262394710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc100700 msgr2=0x7fe1bc100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4262394710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc100700 0x7fe1bc100b20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009900 tx=0x7fe1b802bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 -- 172.21.15.78:0/4262394710 shutdown_connections 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4262394710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc1022b0 0x7fe1bc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4262394710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc1018f0 0x7fe1bc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4262394710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc100700 0x7fe1bc100b20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 -- 172.21.15.78:0/4262394710 >> 172.21.15.78:0/4262394710 conn(0x7fe1bc0fbc70 msgr2=0x7fe1bc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 -- 172.21.15.78:0/4262394710 shutdown_connections 2024-03-16T11:07:03.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.044+0000 7fe1c4377640 1 -- 172.21.15.78:0/4262394710 wait complete. 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 Processor -- start 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 -- start start 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc100700 0x7fe1bc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc1018f0 0x7fe1bc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc1022b0 0x7fe1bc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc069380 con 0x7fe1bc100700 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc069200 con 0x7fe1bc1018f0 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c4377640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc069500 con 0x7fe1bc1022b0 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc100700 0x7fe1bc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc100700 0x7fe1bc197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55070/0 (socket says 172.21.15.78:55070) 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 -- 172.21.15.78:0/4043106043 learned_addr learned my addr 172.21.15.78:0/4043106043 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c1912640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc1018f0 0x7fe1bc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2914640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc1022b0 0x7fe1bc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 -- 172.21.15.78:0/4043106043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc1022b0 msgr2=0x7fe1bc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc1022b0 0x7fe1bc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 -- 172.21.15.78:0/4043106043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc1018f0 msgr2=0x7fe1bc197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc1018f0 0x7fe1bc197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c2113640 1 -- 172.21.15.78:0/4043106043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc19f670 con 0x7fe1bc100700 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.045+0000 7fe1c1912640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc1018f0 0x7fe1bc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.046+0000 7fe1c2113640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc100700 0x7fe1bc197a50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8000c00 tx=0x7fe1b8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.046+0000 7fe1b37fe640 1 -- 172.21.15.78:0/4043106043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8043020 con 0x7fe1bc100700 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.046+0000 7fe1b37fe640 1 -- 172.21.15.78:0/4043106043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1b8004440 con 0x7fe1bc100700 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.046+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc19f900 con 0x7fe1bc100700 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.046+0000 7fe1b37fe640 1 -- 172.21.15.78:0/4043106043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8003c90 con 0x7fe1bc100700 2024-03-16T11:07:03.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.046+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc19fea0 con 0x7fe1bc100700 2024-03-16T11:07:03.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.047+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1bc107a70 con 0x7fe1bc100700 2024-03-16T11:07:03.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.049+0000 7fe1b37fe640 1 -- 172.21.15.78:0/4043106043 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fe1b8046020 con 0x7fe1bc100700 2024-03-16T11:07:03.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.050+0000 7fe1b37fe640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe1a80a2910 0x7fe1a80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.050+0000 7fe1b37fe640 1 -- 172.21.15.78:0/4043106043 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe1b80f0890 con 0x7fe1bc100700 2024-03-16T11:07:03.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.051+0000 7fe1c1912640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe1a80a2910 0x7fe1a80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.051+0000 7fe1c1912640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe1a80a2910 0x7fe1a80a4dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc198f60 tx=0x7fe1ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.059+0000 7fe1b37fe640 1 -- 172.21.15.78:0/4043106043 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1b804d030 con 0x7fe1bc100700 2024-03-16T11:07:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:03 smithi098 ceph-mon[35848]: pgmap v380: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:07:03.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.498+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1bc02d1b0 con 0x7fe1bc100700 2024-03-16T11:07:03.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.499+0000 7fe1b37fe640 1 -- 172.21.15.78:0/4043106043 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7fe1b80bd8e0 con 0x7fe1bc100700 2024-03-16T11:07:03.500 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:07:03.500 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:07:03.500 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:07:03.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe1a80a2910 msgr2=0x7fe1a80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe1a80a2910 0x7fe1a80a4dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc198f60 tx=0x7fe1ac005f90 comp rx=0 tx=0).stop 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc100700 msgr2=0x7fe1bc197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc100700 0x7fe1bc197a50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8000c00 tx=0x7fe1b8004040 comp rx=0 tx=0).stop 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 shutdown_connections 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe1a80a2910 0x7fe1a80a4dd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc1022b0 0x7fe1bc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc1018f0 0x7fe1bc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 --2- 172.21.15.78:0/4043106043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1bc100700 0x7fe1bc197a50 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 >> 172.21.15.78:0/4043106043 conn(0x7fe1bc0fbc70 msgr2=0x7fe1bc104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 shutdown_connections 2024-03-16T11:07:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.502+0000 7fe1c4377640 1 -- 172.21.15.78:0/4043106043 wait complete. 2024-03-16T11:07:03.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:03 smithi078 ceph-mon[32235]: pgmap v380: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:07:03.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T11:07:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:03 smithi153 ceph-mon[35521]: pgmap v380: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:07:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 -- 172.21.15.78:0/2352410294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c680ff850 msgr2=0x7f0c680ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/2352410294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c680ff850 0x7f0c680ffcd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009860 tx=0x7f0c5002bb70 comp rx=0 tx=0).stop 2024-03-16T11:07:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 -- 172.21.15.78:0/2352410294 shutdown_connections 2024-03-16T11:07:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/2352410294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c68100210 0x7f0c681006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/2352410294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c680ff850 0x7f0c680ffcd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/2352410294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c68101d60 0x7f0c68102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 -- 172.21.15.78:0/2352410294 >> 172.21.15.78:0/2352410294 conn(0x7f0c680fb3d0 msgr2=0x7f0c680fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 -- 172.21.15.78:0/2352410294 shutdown_connections 2024-03-16T11:07:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.626+0000 7f0c6cd89640 1 -- 172.21.15.78:0/2352410294 wait complete. 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 Processor -- start 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 -- start start 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c680ff850 0x7f0c68199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c68100210 0x7f0c6819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68101d60 0x7f0c681a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810f520 con 0x7f0c68101d60 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810f3a0 con 0x7f0c680ff850 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.627+0000 7f0c6cd89640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810f6a0 con 0x7f0c68100210 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c680ff850 0x7f0c68199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c680ff850 0x7f0c68199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43076/0 (socket says 172.21.15.78:43076) 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c65d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c68100210 0x7f0c6819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 -- 172.21.15.78:0/4007133361 learned_addr learned my addr 172.21.15.78:0/4007133361 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68101d60 0x7f0c681a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 -- 172.21.15.78:0/4007133361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c68100210 msgr2=0x7f0c6819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c68100210 0x7f0c6819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 -- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68101d60 msgr2=0x7f0c681a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68101d60 0x7f0c681a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 -- 172.21.15.78:0/4007133361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c681a1890 con 0x7f0c680ff850 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6659c640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c680ff850 0x7f0c68199c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800e9f0 tx=0x7f0c5800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c5f7fe640 1 -- 172.21.15.78:0/4007133361 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5800ccc0 con 0x7f0c680ff850 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c5f7fe640 1 -- 172.21.15.78:0/4007133361 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0c580054d0 con 0x7f0c680ff850 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c5f7fe640 1 -- 172.21.15.78:0/4007133361 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58010430 con 0x7f0c680ff850 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c681a1b80 con 0x7f0c680ff850 2024-03-16T11:07:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.628+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c681a20d0 con 0x7f0c680ff850 2024-03-16T11:07:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.631+0000 7f0c5f7fe640 1 -- 172.21.15.78:0/4007133361 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f0c58010610 con 0x7f0c680ff850 2024-03-16T11:07:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.631+0000 7f0c6f979740 1 Processor -- start 2024-03-16T11:07:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6f979740 1 -- start start 2024-03-16T11:07:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7c7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6f979740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558cea724490 con 0x558cea7c0240 2024-03-16T11:07:03.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6f979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558cea6ccc90 con 0x558cea7bf850 2024-03-16T11:07:03.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6f979740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558cea720530 con 0x558cea7c53b0 2024-03-16T11:07:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c65d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7bfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43090/0 (socket says 172.21.15.78:43090) 2024-03-16T11:07:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 -- 172.21.15.78:0/3836806657 learned_addr learned my addr 172.21.15.78:0/3836806657 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c6659c640 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 -- 172.21.15.78:0/3836806657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 msgr2=0x558cea7c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 -- 172.21.15.78:0/3836806657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 msgr2=0x558cea7c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.632+0000 7f0c66d9d640 1 -- 172.21.15.78:0/3836806657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cea787500 con 0x558cea7bf850 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.633+0000 7f0c5f7fe640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0c540a2910 0x7f0c540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.633+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7bfc70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009fd0 tx=0x7f0c600118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eb93a38ca5a083 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.633+0000 7f0c6659c640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0c540a2910 0x7f0c540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.633+0000 7f0c5cff9640 1 -- 172.21.15.78:0/3836806657 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60004d20 con 0x558cea7bf850 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.633+0000 7f0c5cff9640 1 -- 172.21.15.78:0/3836806657 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0c60004050 con 0x558cea7bf850 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.633+0000 7f0c5cff9640 1 -- 172.21.15.78:0/3836806657 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60005280 con 0x558cea7bf850 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.633+0000 7f0c5f7fe640 1 -- 172.21.15.78:0/4007133361 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0c580cee80 con 0x7f0c680ff850 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 -- 172.21.15.78:0/3836806657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 msgr2=0x558cea7bfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7bfc70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009fd0 tx=0x7f0c600118d0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6659c640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0c540a2910 0x7f0c540a4dd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x558cea7c0cb0 tx=0x7f0c58009f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 -- 172.21.15.78:0/3836806657 shutdown_connections 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 --2- 172.21.15.78:0/3836806657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7bfc70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 -- 172.21.15.78:0/3836806657 >> 172.21.15.78:0/3836806657 conn(0x558cea7badc0 msgr2=0x558cea7bb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 -- 172.21.15.78:0/3836806657 shutdown_connections 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 -- 172.21.15.78:0/3836806657 wait complete. 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 -- start start 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.634+0000 7f0c6f979740 1 Processor -- start 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c6f979740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558cea7c8360 con 0x558cea7c0240 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c6f979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558cea6e0230 con 0x558cea7bf850 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c6f979740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558cea7c84e0 con 0x558cea7c53b0 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7c9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43092/0 (socket says 172.21.15.78:43092) 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1151499222 learned_addr learned my addr 172.21.15.78:0/1151499222 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c65d9b640 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c6f979740 10 client.0 _ll_register_callbacks cb 0x558cea71c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1151499222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 msgr2=0x558cea7d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7d12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1151499222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 msgr2=0x558cea7ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1151499222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cea6ef100 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7c9d60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009b00 tx=0x7f0c60009a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51222ce3f0daffba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.635+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1151499222 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6002c020 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1151499222 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0c6000f4e0 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c60004050 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1151499222 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0c6000f6e0 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 10 client.25508 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558cea4383f0 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1151499222 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0c60004050 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1151499222 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0c60028550 con 0x558cea7bf850 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 10 client.25508 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 10 client.25508 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:07:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 20 client.25508 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea7de170 0x558cea7e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 4 client.25508 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cea7319d0 con 0x558cea7de170 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.636+0000 7f0c65d9b640 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea7de170 0x558cea7e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.637+0000 7f0c4dffb640 10 client.25508 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.637+0000 7f0c65d9b640 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea7de170 0x558cea7e05e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=262c15c997e51333 server_cookie=daff3de16a75b8cf in_seq=0 out_seq=0 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.637+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1151499222 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558cea7319d0 con 0x558cea7de170 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.637+0000 7f0c4dffb640 10 client.25508 handle_command_reply: tid=0 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.637+0000 7f0c6f979740 1 client.25508 shutdown 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 msgr2=0x558cea7c9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7c9d60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009b00 tx=0x7f0c60009a50 comp rx=0 tx=0).stop 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 shutdown_connections 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea7de170 0x558cea7e05e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7c53b0 0x558cea7d12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c4dffb640 0 client.25508 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c0240 0x558cea7ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1151499222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7bf850 0x558cea7c9d60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 >> 172.21.15.78:0/1151499222 conn(0x558cea7badc0 msgr2=0x558cea7b9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 shutdown_connections 2024-03-16T11:07:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 1 -- 172.21.15.78:0/1151499222 wait complete. 2024-03-16T11:07:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.638+0000 7f0c6f979740 20 client.25508 trim_cache size 0 max 16384 2024-03-16T11:07:03.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 Processor -- start 2024-03-16T11:07:03.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 -- start start 2024-03-16T11:07:03.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7bb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0c681a9360 con 0x558cea7c53b0 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c681a6c30 con 0x558cea7c0240 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c6f979740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c681a4a60 con 0x558cea7bf850 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c65d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7bb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c65d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7d1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43096/0 (socket says 172.21.15.78:43096) 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7bb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54124/0 (socket says 172.21.15.78:54124) 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.651+0000 7f0c65d9b640 1 -- 172.21.15.78:0/1817183913 learned_addr learned my addr 172.21.15.78:0/1817183913 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1817183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 msgr2=0x558cea7d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7d1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1817183913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 msgr2=0x558cea7b9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7b9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1817183913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cea5027d0 con 0x558cea7bf850 2024-03-16T11:07:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c65d9b640 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7bb1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c60028f80 tx=0x7f0c60000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7d4a0517183dc68 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c4ffff640 1 -- 172.21.15.78:0/1817183913 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6002c070 con 0x558cea7bf850 2024-03-16T11:07:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c4ffff640 1 -- 172.21.15.78:0/1817183913 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0c60009b30 con 0x558cea7bf850 2024-03-16T11:07:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.652+0000 7f0c4ffff640 1 -- 172.21.15.78:0/1817183913 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6000f880 con 0x558cea7bf850 2024-03-16T11:07:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- 172.21.15.78:0/1817183913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 msgr2=0x558cea7bb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7bb1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c60028f80 tx=0x7f0c60000f40 comp rx=0 tx=0).stop 2024-03-16T11:07:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- 172.21.15.78:0/1817183913 shutdown_connections 2024-03-16T11:07:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1817183913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7bb1a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- 172.21.15.78:0/1817183913 >> 172.21.15.78:0/1817183913 conn(0x558cea7badc0 msgr2=0x558cea7c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- 172.21.15.78:0/1817183913 shutdown_connections 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- 172.21.15.78:0/1817183913 wait complete. 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- start start 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 Processor -- start 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0c681a50f0 con 0x558cea7c53b0 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c681a6560 con 0x558cea7c0240 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.653+0000 7f0c6f979740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c681a57c0 con 0x558cea7bf850 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c6f979740 10 client.0 _ll_register_callbacks cb 0x7f0c600093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55120/0 (socket says 172.21.15.78:55120) 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1580368441 learned_addr learned my addr 172.21.15.78:0/1580368441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c65d9b640 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1580368441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 msgr2=0x558cea7ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 msgr2=0x558cea7cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 -- 172.21.15.78:0/1580368441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cea7d0210 con 0x558cea7c53b0 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c66d9d640 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7cfc20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0c60005fd0 tx=0x7f0c60005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb04a4a36f9b1561 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1580368441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6002c030 con 0x558cea7c53b0 2024-03-16T11:07:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1580368441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0c60028650 con 0x558cea7c53b0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.654+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c60009b30 con 0x558cea7c53b0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1580368441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0c6000f880 con 0x558cea7c53b0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 10 client.15681 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558cea7d1510 con 0x558cea7c53b0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1580368441 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0c60004950 con 0x558cea7c53b0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1580368441 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0c60005040 con 0x558cea7c53b0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 10 client.15681 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 10 client.15681 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 20 client.15681 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea844ab0 0x558cea846ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 4 client.15681 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0c60009b30 con 0x558cea844ab0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.655+0000 7f0c65d9b640 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea844ab0 0x558cea846ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c4dffb640 10 client.15681 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c65d9b640 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea844ab0 0x558cea846ed0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=25af85ec3645242c server_cookie=35318573b79ba965 in_seq=0 out_seq=0 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c4dffb640 1 -- 172.21.15.78:0/1580368441 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0c60009b30 con 0x558cea844ab0 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c4dffb640 10 client.15681 handle_command_reply: tid=0 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 client.15681 shutdown 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 msgr2=0x558cea7cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7cfc20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0c60005fd0 tx=0x7f0c60005cf0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 shutdown_connections 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x558cea844ab0 0x558cea846ed0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cea7c53b0 0x558cea7cfc20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558cea7c0240 0x558cea7cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 --2- 172.21.15.78:0/1580368441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558cea7bf850 0x558cea7ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 >> 172.21.15.78:0/1580368441 conn(0x558cea7badc0 msgr2=0x558cea7bb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c4dffb640 0 client.15681 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 shutdown_connections 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 1 -- 172.21.15.78:0/1580368441 wait complete. 2024-03-16T11:07:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.656+0000 7f0c6f979740 20 client.15681 trim_cache size 0 max 16384 2024-03-16T11:07:03.662 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:07:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0c540a2910 msgr2=0x7f0c540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0c540a2910 0x7f0c540a4dd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x558cea7c0cb0 tx=0x7f0c58009f50 comp rx=0 tx=0).stop 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c680ff850 msgr2=0x7f0c68199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c680ff850 0x7f0c68199c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800e9f0 tx=0x7f0c5800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 shutdown_connections 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68101d60 0x7f0c681a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0c540a2910 0x7f0c540a4dd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c68100210 0x7f0c6819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 --2- 172.21.15.78:0/4007133361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c680ff850 0x7f0c68199c80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 >> 172.21.15.78:0/4007133361 conn(0x7f0c680fb3d0 msgr2=0x7f0c680fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 shutdown_connections 2024-03-16T11:07:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.657+0000 7f0c6cd89640 1 -- 172.21.15.78:0/4007133361 wait complete. 2024-03-16T11:07:03.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T11:07:03.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3734528436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e4102290 msgr2=0x7f39e4108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3734528436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e4102290 0x7f39e4108b20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009ca0 tx=0x7f39dc02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3734528436 shutdown_connections 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3734528436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e4102290 0x7f39e4108b20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3734528436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e41018d0 0x7f39e4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3734528436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41006e0 0x7f39e4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3734528436 >> 172.21.15.78:0/3734528436 conn(0x7f39e40fbc70 msgr2=0x7f39e40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3734528436 shutdown_connections 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.775+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3734528436 wait complete. 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 Processor -- start 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 -- start start 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e41006e0 0x7f39e41968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e41018d0 0x7f39e4196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e41006e0 0x7f39e41968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e4102290 0x7f39e419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f39e410c210 con 0x7f39e41006e0 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39e410c090 con 0x7f39e4102290 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.776+0000 7f39e9ca1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39e410c390 con 0x7f39e41018d0 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e41006e0 0x7f39e41968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55138/0 (socket says 172.21.15.78:55138) 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 -- 172.21.15.78:0/3766342857 learned_addr learned my addr 172.21.15.78:0/3766342857 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e41018d0 0x7f39e4196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e3fff640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e4102290 0x7f39e419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 -- 172.21.15.78:0/3766342857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e41018d0 msgr2=0x7f39e4196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e41018d0 0x7f39e4196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 -- 172.21.15.78:0/3766342857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e4102290 msgr2=0x7f39e419de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e4102290 0x7f39e419de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 -- 172.21.15.78:0/3766342857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e419e500 con 0x7f39e41006e0 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e41018d0 0x7f39e4196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e37fe640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e41006e0 0x7f39e41968e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f39d400e9c0 tx=0x7f39d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e0ff9640 1 -- 172.21.15.78:0/3766342857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d400cde0 con 0x7f39e41006e0 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e419e7f0 con 0x7f39e41006e0 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e419ed90 con 0x7f39e41006e0 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.777+0000 7f39e0ff9640 1 -- 172.21.15.78:0/3766342857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39d40108e0 con 0x7f39e41006e0 2024-03-16T11:07:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.778+0000 7f39e0ff9640 1 -- 172.21.15.78:0/3766342857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40187a0 con 0x7f39e41006e0 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.780+0000 7f39e0ff9640 1 -- 172.21.15.78:0/3766342857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f39d401f020 con 0x7f39e41006e0 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39e0ff9640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f39cc0a2910 0x7f39cc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39e0ff9640 1 -- 172.21.15.78:0/3766342857 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f39d40ced80 con 0x7f39e41006e0 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 Processor -- start 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f39cc0a2910 0x7f39cc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 -- start start 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd42ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42efb60 0x55afd42effe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd42fbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55afd4254490 con 0x55afd42f0ed0 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afd41fcc90 con 0x55afd42efb60 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39ec96c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55afd4250530 con 0x55afd42f9b70 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd42ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd42ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55148/0 (socket says 172.21.15.78:55148) 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39e37fe640 1 -- 172.21.15.78:0/2890021908 learned_addr learned my addr 172.21.15.78:0/2890021908 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.781+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f39cc0a2910 0x7f39cc0a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f39e4197df0 tx=0x7f39d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e3fff640 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd42fbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42efb60 0x55afd42effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e37fe640 1 -- 172.21.15.78:0/2890021908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 msgr2=0x55afd42fbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd42fbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e37fe640 1 -- 172.21.15.78:0/2890021908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42efb60 msgr2=0x55afd42effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42efb60 0x55afd42effe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e37fe640 1 -- 172.21.15.78:0/2890021908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd42b7500 con 0x55afd42f0ed0 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd42ef620 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f39d4097f50 tx=0x7f39d409b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d3adca6f21e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39c9ffb640 1 -- 172.21.15.78:0/2890021908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40a3d20 con 0x55afd42f0ed0 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39c9ffb640 1 -- 172.21.15.78:0/2890021908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39d4097bf0 con 0x55afd42f0ed0 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.782+0000 7f39c9ffb640 1 -- 172.21.15.78:0/2890021908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d409ac30 con 0x55afd42f0ed0 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 -- 172.21.15.78:0/2890021908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 msgr2=0x55afd42ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd42ef620 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f39d4097f50 tx=0x7f39d409b600 comp rx=0 tx=0).stop 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 -- 172.21.15.78:0/2890021908 shutdown_connections 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd42fbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42efb60 0x55afd42effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2890021908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd42ef620 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 -- 172.21.15.78:0/2890021908 >> 172.21.15.78:0/2890021908 conn(0x55afd42eadc0 msgr2=0x55afd42eb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 -- 172.21.15.78:0/2890021908 shutdown_connections 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 -- 172.21.15.78:0/2890021908 wait complete. 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 -- start start 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.783+0000 7f39ec96c740 1 Processor -- start 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd42fe160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42f0ed0 0x55afd42fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd43056e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39ec96c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55afd42fc7a0 con 0x55afd42efb60 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39ec96c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afd4210230 con 0x55afd42f0ed0 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39ec96c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55afd42fc920 con 0x55afd42f9b70 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd42fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd42fe160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55150/0 (socket says 172.21.15.78:55150) 2024-03-16T11:07:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 -- 172.21.15.78:0/2934687087 learned_addr learned my addr 172.21.15.78:0/2934687087 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42f0ed0 0x55afd42fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e3fff640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd43056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39ec96c740 10 client.0 _ll_register_callbacks cb 0x55afd424c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 -- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 msgr2=0x55afd43056e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd43056e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 -- 172.21.15.78:0/2934687087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42f0ed0 msgr2=0x55afd42fe6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42f0ed0 0x55afd42fe6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 -- 172.21.15.78:0/2934687087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd421f100 con 0x55afd42efb60 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42f0ed0 0x55afd42fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd42fe160 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f39d4000be0 tx=0x7f39d40106c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93545e3b429e125e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.784+0000 7f39c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:03.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2934687087 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40b8020 con 0x55afd42efb60 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2934687087 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f39d409a9d0 con 0x55afd42efb60 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39d4097bf0 con 0x55afd42efb60 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2934687087 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f39d40a3d20 con 0x55afd42efb60 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 10 client.15693 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55afd3f683f0 con 0x55afd42efb60 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2934687087 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39d4097bf0 con 0x55afd42efb60 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2934687087 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f39d4018940 con 0x55afd42efb60 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 10 client.15693 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 10 client.15693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 20 client.15693 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd4309b90 0x55afd430bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 4 client.15693 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.785+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afd42619d0 con 0x55afd4309b90 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.786+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd4309b90 0x55afd430bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.786+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd4309b90 0x55afd430bf80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e81053596f62a48c server_cookie=b6d4114a5e28e7c3 in_seq=0 out_seq=0 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.786+0000 7f39c2ffd640 10 client.15693 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2934687087 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afd42619d0 con 0x55afd4309b90 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39c2ffd640 10 client.15693 handle_command_reply: tid=0 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 client.15693 shutdown 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 msgr2=0x55afd42fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd42fe160 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f39d4000be0 tx=0x7f39d40106c0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 shutdown_connections 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd4309b90 0x55afd430bf80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f9b70 0x55afd43056e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42f0ed0 0x55afd42fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2934687087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd42fe160 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 >> 172.21.15.78:0/2934687087 conn(0x55afd42eadc0 msgr2=0x55afd42e9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39c2ffd640 0 client.15693 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 shutdown_connections 2024-03-16T11:07:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 1 -- 172.21.15.78:0/2934687087 wait complete. 2024-03-16T11:07:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.787+0000 7f39ec96c740 20 client.15693 trim_cache size 0 max 16384 2024-03-16T11:07:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 Processor -- start 2024-03-16T11:07:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 -- start start 2024-03-16T11:07:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd42fe480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd43056a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f0ed0 0x55afd42e9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f39d0010ca0 con 0x55afd42efb60 2024-03-16T11:07:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39d000e570 con 0x55afd42eadc0 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39ec96c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39d000c3a0 con 0x55afd42f0ed0 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39e2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd43056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd42fe480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39e2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd43056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55166/0 (socket says 172.21.15.78:55166) 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39e37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd42fe480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43150/0 (socket says 172.21.15.78:43150) 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39e2ffd640 1 -- 172.21.15.78:0/2964830052 learned_addr learned my addr 172.21.15.78:0/2964830052 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39e3fff640 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f0ed0 0x55afd42e9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.800+0000 7f39e37fe640 1 -- 172.21.15.78:0/2964830052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f0ed0 msgr2=0x55afd42e9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f0ed0 0x55afd42e9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39e37fe640 1 -- 172.21.15.78:0/2964830052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 msgr2=0x55afd43056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd43056a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39e37fe640 1 -- 172.21.15.78:0/2964830052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd40327d0 con 0x55afd42eadc0 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd42fe480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f39d40095f0 tx=0x7f39d400efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54ea8d9c124520cb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39c97fa640 1 -- 172.21.15.78:0/2964830052 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40b8070 con 0x55afd42eadc0 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39c97fa640 1 -- 172.21.15.78:0/2964830052 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39d4010730 con 0x55afd42eadc0 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.801+0000 7f39c97fa640 1 -- 172.21.15.78:0/2964830052 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40a3900 con 0x55afd42eadc0 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- 172.21.15.78:0/2964830052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 msgr2=0x55afd42fe480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd42fe480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f39d40095f0 tx=0x7f39d400efb0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- 172.21.15.78:0/2964830052 shutdown_connections 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42f0ed0 0x55afd42e9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42efb60 0x55afd43056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2964830052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd42fe480 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- 172.21.15.78:0/2964830052 >> 172.21.15.78:0/2964830052 conn(0x55afd42ef780 msgr2=0x55afd42fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- 172.21.15.78:0/2964830052 shutdown_connections 2024-03-16T11:07:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- 172.21.15.78:0/2964830052 wait complete. 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- start start 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 Processor -- start 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd43029e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42efb60 0x55afd4302f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd4303f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f39d000ca30 con 0x55afd42f0ed0 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39d000dea0 con 0x55afd42eadc0 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39d000d100 con 0x55afd42efb60 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.802+0000 7f39ec96c740 10 client.0 _ll_register_callbacks cb 0x7f39d40bb050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd4303f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd4303f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55180/0 (socket says 172.21.15.78:55180) 2024-03-16T11:07:03.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42efb60 0x55afd4302f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 -- 172.21.15.78:0/2422029129 learned_addr learned my addr 172.21.15.78:0/2422029129 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd43029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 -- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42efb60 msgr2=0x55afd4302f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42efb60 0x55afd4302f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 -- 172.21.15.78:0/2422029129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 msgr2=0x55afd43029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd43029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 -- 172.21.15.78:0/2422029129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd4304520 con 0x55afd42f0ed0 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd43029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e2ffd640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42efb60 0x55afd4302f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39e3fff640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd4303f30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f39dc032550 tx=0x7f39dc032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1c337c628267af server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2422029129 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc039040 con 0x55afd42f0ed0 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2422029129 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f39dc032910 con 0x55afd42f0ed0 2024-03-16T11:07:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.803+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39d4010730 con 0x55afd42f0ed0 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2422029129 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f39dc003b80 con 0x55afd42f0ed0 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 10 client.15699 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55afd43058c0 con 0x55afd42f0ed0 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2422029129 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39d4010730 con 0x55afd42f0ed0 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2422029129 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f39dc0344f0 con 0x55afd42f0ed0 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 10 client.15699 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 10 client.15699 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 20 client.15699 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd42f9b70 0x55afd4372400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 4 client.15699 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afd425bd20 con 0x55afd42f9b70 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.804+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd42f9b70 0x55afd4372400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39c2ffd640 10 client.15699 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39e37fe640 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd42f9b70 0x55afd4372400 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e13f721c93e331c server_cookie=8ea6509557522995 in_seq=0 out_seq=0 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39c2ffd640 1 -- 172.21.15.78:0/2422029129 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afd425bd20 con 0x55afd42f9b70 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39c2ffd640 10 client.15699 handle_command_reply: tid=0 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 client.15699 shutdown 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 msgr2=0x55afd4303f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd4303f30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f39dc032550 tx=0x7f39dc032820 comp rx=0 tx=0).stop 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 shutdown_connections 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55afd42f9b70 0x55afd4372400 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afd42f0ed0 0x55afd4303f30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39c2ffd640 0 client.15699 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afd42efb60 0x55afd4302f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 --2- 172.21.15.78:0/2422029129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afd42eadc0 0x55afd43029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 >> 172.21.15.78:0/2422029129 conn(0x55afd42ef780 msgr2=0x55afd42fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 shutdown_connections 2024-03-16T11:07:03.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 1 -- 172.21.15.78:0/2422029129 wait complete. 2024-03-16T11:07:03.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.805+0000 7f39ec96c740 20 client.15699 trim_cache size 0 max 16384 2024-03-16T11:07:03.813 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:07:03.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f39cc0a2910 msgr2=0x7f39cc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f39cc0a2910 0x7f39cc0a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f39e4197df0 tx=0x7f39d8005f90 comp rx=0 tx=0).stop 2024-03-16T11:07:03.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e41006e0 msgr2=0x7f39e41968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e41006e0 0x7f39e41968e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f39d400e9c0 tx=0x7f39d400ee80 comp rx=0 tx=0).stop 2024-03-16T11:07:03.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 shutdown_connections 2024-03-16T11:07:03.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e4102290 0x7f39e419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f39cc0a2910 0x7f39cc0a4dd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e41018d0 0x7f39e4196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 --2- 172.21.15.78:0/3766342857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39e41006e0 0x7f39e41968e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 >> 172.21.15.78:0/3766342857 conn(0x7f39e40fbc70 msgr2=0x7f39e4104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 shutdown_connections 2024-03-16T11:07:03.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.806+0000 7f39e9ca1640 1 -- 172.21.15.78:0/3766342857 wait complete. 2024-03-16T11:07:03.818 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T11:07:03.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 -- 172.21.15.78:0/1163371313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681033a0 msgr2=0x7f6068109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 --2- 172.21.15.78:0/1163371313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681033a0 0x7f6068109c30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6060009be0 tx=0x7f606002bef0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 -- 172.21.15.78:0/1163371313 shutdown_connections 2024-03-16T11:07:03.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 --2- 172.21.15.78:0/1163371313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681033a0 0x7f6068109c30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 --2- 172.21.15.78:0/1163371313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681029e0 0x7f6068102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 --2- 172.21.15.78:0/1163371313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60681017f0 0x7f6068101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 -- 172.21.15.78:0/1163371313 >> 172.21.15.78:0/1163371313 conn(0x7f60680fcd60 msgr2=0x7f60680ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 -- 172.21.15.78:0/1163371313 shutdown_connections 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.923+0000 7f606ee80640 1 -- 172.21.15.78:0/1163371313 wait complete. 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 Processor -- start 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 -- start start 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681017f0 0x7f6068075540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60681029e0 0x7f6068073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681033a0 0x7f6068074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f606810d300 con 0x7f60681033a0 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f606810d180 con 0x7f60681029e0 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606ee80640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f606810d480 con 0x7f60681017f0 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606cc1c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681017f0 0x7f6068075540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606d41d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681033a0 0x7f6068074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606d41d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681033a0 0x7f6068074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55200/0 (socket says 172.21.15.78:55200) 2024-03-16T11:07:03.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606d41d640 1 -- 172.21.15.78:0/3105354887 learned_addr learned my addr 172.21.15.78:0/3105354887 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f6067fff640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60681029e0 0x7f6068073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f6067fff640 1 -- 172.21.15.78:0/3105354887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681017f0 msgr2=0x7f6068075540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f6067fff640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681017f0 0x7f6068075540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f6067fff640 1 -- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681033a0 msgr2=0x7f6068074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f6067fff640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681033a0 0x7f6068074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f6067fff640 1 -- 172.21.15.78:0/3105354887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60680749d0 con 0x7f60681029e0 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.924+0000 7f606d41d640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681033a0 0x7f6068074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.925+0000 7f6067fff640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60681029e0 0x7f6068073bd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6054005fd0 tx=0x7f605400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.925+0000 7f6065ffb640 1 -- 172.21.15.78:0/3105354887 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605400cd00 con 0x7f60681029e0 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.925+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60681a1b50 con 0x7f60681029e0 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.925+0000 7f6065ffb640 1 -- 172.21.15.78:0/3105354887 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60540035f0 con 0x7f60681029e0 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.925+0000 7f6065ffb640 1 -- 172.21.15.78:0/3105354887 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054010690 con 0x7f60681029e0 2024-03-16T11:07:03.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.925+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60681a20c0 con 0x7f60681029e0 2024-03-16T11:07:03.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.927+0000 7f6065ffb640 1 -- 172.21.15.78:0/3105354887 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f60540037a0 con 0x7f60681029e0 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.928+0000 7f6065ffb640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60580a2910 0x7f60580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.928+0000 7f6065ffb640 1 -- 172.21.15.78:0/3105354887 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f60540cedc0 con 0x7f60681029e0 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.928+0000 7f606cc1c640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60580a2910 0x7f60580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.928+0000 7f606cc1c640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60580a2910 0x7f60580a4dd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6068102840 tx=0x7f605c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 Processor -- start 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 -- start start 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c0fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c12840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561671b6f490 con 0x561671c0b240 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561671b17c90 con 0x561671c0a850 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6071a5f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561671b6b530 con 0x561671c103b0 2024-03-16T11:07:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6067fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c0fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6067fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c0fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55212/0 (socket says 172.21.15.78:55212) 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f6067fff640 1 -- 172.21.15.78:0/3772085920 learned_addr learned my addr 172.21.15.78:0/3772085920 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f606cc1c640 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f606d41d640 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c12840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f606d41d640 1 -- 172.21.15.78:0/3772085920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 msgr2=0x561671c0ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f606d41d640 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f606d41d640 1 -- 172.21.15.78:0/3772085920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 msgr2=0x561671c0fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f606d41d640 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c0fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.929+0000 7f606d41d640 1 -- 172.21.15.78:0/3772085920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561671bd2500 con 0x561671c103b0 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.930+0000 7f6067fff640 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c0fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.930+0000 7f606cc1c640 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.930+0000 7f606d41d640 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c12840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6060006010 tx=0x7f6060005c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec642e2f189b2b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.930+0000 7f6052ffd640 1 -- 172.21.15.78:0/3772085920 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6060043030 con 0x561671c103b0 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.930+0000 7f6052ffd640 1 -- 172.21.15.78:0/3772085920 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6060003970 con 0x561671c103b0 2024-03-16T11:07:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.930+0000 7f6052ffd640 1 -- 172.21.15.78:0/3772085920 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606002cb80 con 0x561671c103b0 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 -- 172.21.15.78:0/3772085920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 msgr2=0x561671c12840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c12840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6060006010 tx=0x7f6060005c60 comp rx=0 tx=0).stop 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 -- 172.21.15.78:0/3772085920 shutdown_connections 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c12840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c0fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 --2- 172.21.15.78:0/3772085920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 -- 172.21.15.78:0/3772085920 >> 172.21.15.78:0/3772085920 conn(0x561671c05dc0 msgr2=0x561671c061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 -- 172.21.15.78:0/3772085920 shutdown_connections 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 -- 172.21.15.78:0/3772085920 wait complete. 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 -- start start 2024-03-16T11:07:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.931+0000 7f6071a5f740 1 Processor -- start 2024-03-16T11:07:03.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c14cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c15210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6071a5f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561671c13300 con 0x561671c103b0 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6071a5f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561671b2b230 con 0x561671c0a850 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6071a5f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561671c13480 con 0x561671c0b240 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6071a5f740 10 client.0 _ll_register_callbacks cb 0x561671b67010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c15210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f606d41d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c15210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54216/0 (socket says 172.21.15.78:54216) 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 -- 172.21.15.78:0/1850453306 learned_addr learned my addr 172.21.15.78:0/1850453306 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f606cc1c640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c14cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 -- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 msgr2=0x561671c14cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c14cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 -- 172.21.15.78:0/1850453306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 msgr2=0x561671c1c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f606cc1c640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c14cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f6067fff640 1 -- 172.21.15.78:0/1850453306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561671b3a100 con 0x561671c0b240 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.932+0000 7f606d41d640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6067fff640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c15210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f605400c6a0 tx=0x7f605409bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd329d88d763f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6047fff640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6047fff640 1 -- 172.21.15.78:0/1850453306 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60540b4060 con 0x561671c0b240 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6047fff640 1 -- 172.21.15.78:0/1850453306 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f60540b4200 con 0x561671c0b240 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6060003970 con 0x561671c0b240 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6047fff640 1 -- 172.21.15.78:0/1850453306 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f60540b9050 con 0x561671c0b240 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6071a5f740 10 client.25531 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5616718833f0 con 0x561671c0b240 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.933+0000 7f6047fff640 1 -- 172.21.15.78:0/1850453306 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6060003970 con 0x561671c0b240 2024-03-16T11:07:03.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.934+0000 7f6047fff640 1 -- 172.21.15.78:0/1850453306 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f605409aa50 con 0x561671c0b240 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.934+0000 7f6071a5f740 10 client.25531 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.934+0000 7f6071a5f740 10 client.25531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.934+0000 7f6071a5f740 20 client.25531 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.934+0000 7f6071a5f740 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c22f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.934+0000 7f6071a5f740 4 client.25531 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.934+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561671b7c9d0 con 0x561671c20a80 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.935+0000 7f606cc1c640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c22f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.935+0000 7f606cc1c640 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c22f20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e0e39feb7cb03aa server_cookie=7c5cafc0e0bc5899 in_seq=0 out_seq=0 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.935+0000 7f6047fff640 10 client.25531 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6047fff640 1 -- 172.21.15.78:0/1850453306 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561671b7c9d0 con 0x561671c20a80 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6047fff640 10 client.25531 handle_command_reply: tid=0 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 client.25531 shutdown 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 msgr2=0x561671c15210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c15210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f605400c6a0 tx=0x7f605409bd00 comp rx=0 tx=0).stop 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 shutdown_connections 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c22f20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6047fff640 0 client.25531 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c15210 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 --2- 172.21.15.78:0/1850453306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c14cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 >> 172.21.15.78:0/1850453306 conn(0x561671c05dc0 msgr2=0x561671c04fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 shutdown_connections 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 1 -- 172.21.15.78:0/1850453306 wait complete. 2024-03-16T11:07:03.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.936+0000 7f6071a5f740 20 client.25531 trim_cache size 0 max 16384 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.949+0000 7f6071a5f740 1 Processor -- start 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.949+0000 7f6071a5f740 1 -- start start 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c1c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c04750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6071a5f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f60681ad670 con 0x561671c0b240 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6071a5f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60681aaf40 con 0x561671c0a850 2024-03-16T11:07:03.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6071a5f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60681a8d70 con 0x561671c103b0 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6067fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c1c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6067fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c1c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55234/0 (socket says 172.21.15.78:55234) 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6067fff640 1 -- 172.21.15.78:0/184911835 learned_addr learned my addr 172.21.15.78:0/184911835 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f606cc1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f606d41d640 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c04750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f606cc1c640 1 -- 172.21.15.78:0/184911835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 msgr2=0x561671c04750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f606cc1c640 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c04750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f606cc1c640 1 -- 172.21.15.78:0/184911835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 msgr2=0x561671c1c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f606cc1c640 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c1c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f606cc1c640 1 -- 172.21.15.78:0/184911835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56167194d7d0 con 0x561671c0a850 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.950+0000 7f6067fff640 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c1c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.951+0000 7f606cc1c640 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0bb60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x561671c15b70 tx=0x7f605c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c217bf65c1d19b6b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.951+0000 7f60527fc640 1 -- 172.21.15.78:0/184911835 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c02b430 con 0x561671c0a850 2024-03-16T11:07:03.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.951+0000 7f60527fc640 1 -- 172.21.15.78:0/184911835 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f605c02b5d0 con 0x561671c0a850 2024-03-16T11:07:03.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.951+0000 7f60527fc640 1 -- 172.21.15.78:0/184911835 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c003c50 con 0x561671c0a850 2024-03-16T11:07:03.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 -- 172.21.15.78:0/184911835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 msgr2=0x561671c0bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0bb60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x561671c15b70 tx=0x7f605c014000 comp rx=0 tx=0).stop 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 -- 172.21.15.78:0/184911835 shutdown_connections 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c103b0 0x561671c04750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c0b240 0x561671c1c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 --2- 172.21.15.78:0/184911835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c0bb60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 -- 172.21.15.78:0/184911835 >> 172.21.15.78:0/184911835 conn(0x561671c05dc0 msgr2=0x561671c14cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 -- 172.21.15.78:0/184911835 shutdown_connections 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 -- 172.21.15.78:0/184911835 wait complete. 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 -- start start 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.952+0000 7f6071a5f740 1 Processor -- start 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c19300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c19840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6071a5f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f606cc1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c19300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f606cc1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c19300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43236/0 (socket says 172.21.15.78:43236) 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6067fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c19840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f606d41d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6067fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c19840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54232/0 (socket says 172.21.15.78:54232) 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f606d41d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55244/0 (socket says 172.21.15.78:55244) 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f606cc1c640 1 -- 172.21.15.78:0/2376827609 learned_addr learned my addr 172.21.15.78:0/2376827609 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f60681a9400 con 0x561671c103b0 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60681aa870 con 0x561671c0a850 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.953+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60681a9ad0 con 0x561671c0b240 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6071a5f740 10 client.0 _ll_register_callbacks cb 0x7f60540cc6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f606cc1c640 1 -- 172.21.15.78:0/2376827609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 msgr2=0x561671c19840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f606cc1c640 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c19840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f606cc1c640 1 -- 172.21.15.78:0/2376827609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 msgr2=0x561671c1a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f606cc1c640 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f606cc1c640 1 -- 172.21.15.78:0/2376827609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561671c1ae40 con 0x561671c0a850 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f606cc1c640 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c19300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f605c009460 tx=0x7f605c02b550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a88cd2eb48a5f302 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6047fff640 10 client.25550 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f605c02b5d0 con 0x561671c0a850 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6047fff640 1 -- 172.21.15.78:0/2376827609 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c03b030 con 0x561671c0a850 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6047fff640 1 -- 172.21.15.78:0/2376827609 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f605c032040 con 0x561671c0a850 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6047fff640 1 -- 172.21.15.78:0/2376827609 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f605c005040 con 0x561671c0a850 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6071a5f740 10 client.25550 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.954+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=22}) -- 0x561671c1c0e0 con 0x561671c0a850 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6047fff640 1 -- 172.21.15.78:0/2376827609 <== mon.2 v2:172.21.15.98:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f605c0051e0 con 0x561671c0a850 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6071a5f740 10 client.25550 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6071a5f740 10 client.25550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6071a5f740 20 client.25550 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6071a5f740 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c89040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6071a5f740 4 client.25550 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f605c02b5d0 con 0x561671c20a80 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.955+0000 7f6067fff640 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c89040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6047fff640 10 client.25550 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6067fff640 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c89040 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41875a02b4309609 server_cookie=360cefd03e79140 in_seq=0 out_seq=0 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6047fff640 1 -- 172.21.15.78:0/2376827609 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f605c02b5d0 con 0x561671c20a80 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6047fff640 10 client.25550 handle_command_reply: tid=0 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 client.25550 shutdown 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 msgr2=0x561671c19300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c19300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f605c009460 tx=0x7f605c02b550 comp rx=0 tx=0).stop 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 shutdown_connections 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x561671c20a80 0x561671c89040 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6047fff640 0 client.25550 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561671c103b0 0x561671c1a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561671c0b240 0x561671c19840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 --2- 172.21.15.78:0/2376827609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561671c0a850 0x561671c19300 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 >> 172.21.15.78:0/2376827609 conn(0x561671c05dc0 msgr2=0x561671c14cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 shutdown_connections 2024-03-16T11:07:03.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 1 -- 172.21.15.78:0/2376827609 wait complete. 2024-03-16T11:07:03.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.956+0000 7f6071a5f740 20 client.25550 trim_cache size 0 max 16384 2024-03-16T11:07:03.962 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60580a2910 msgr2=0x7f60580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60580a2910 0x7f60580a4dd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6068102840 tx=0x7f605c005f90 comp rx=0 tx=0).stop 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60681029e0 msgr2=0x7f6068073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60681029e0 0x7f6068073bd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6054005fd0 tx=0x7f605400ee50 comp rx=0 tx=0).stop 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 shutdown_connections 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60681033a0 0x7f6068074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60681029e0 0x7f6068073bd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f60580a2910 0x7f60580a4dd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 --2- 172.21.15.78:0/3105354887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60681017f0 0x7f6068075540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:03.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 >> 172.21.15.78:0/3105354887 conn(0x7f60680fcd60 msgr2=0x7f6068105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:03.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 shutdown_connections 2024-03-16T11:07:03.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:03.957+0000 7f606ee80640 1 -- 172.21.15.78:0/3105354887 wait complete. 2024-03-16T11:07:04.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:04 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4043106043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:07:04.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:04 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4043106043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:07:04.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:04 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4043106043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:07:04.969 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T11:07:04.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:07:04.970 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 -- 172.21.15.78:0/2225278104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f4105370 msgr2=0x7f96f41057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 --2- 172.21.15.78:0/2225278104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f4105370 0x7f96f41057f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009ca0 tx=0x7f96ec02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 -- 172.21.15.78:0/2225278104 shutdown_connections 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 --2- 172.21.15.78:0/2225278104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f4105370 0x7f96f41057f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 --2- 172.21.15.78:0/2225278104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f41029c0 0x7f96f4104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 --2- 172.21.15.78:0/2225278104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96f4100050 0x7f96f4102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 -- 172.21.15.78:0/2225278104 >> 172.21.15.78:0/2225278104 conn(0x7f96f40fbc70 msgr2=0x7f96f40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 -- 172.21.15.78:0/2225278104 shutdown_connections 2024-03-16T11:07:05.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.081+0000 7f96f8929640 1 -- 172.21.15.78:0/2225278104 wait complete. 2024-03-16T11:07:05.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 Processor -- start 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 -- start start 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f4100050 0x7f96f419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f41029c0 0x7f96f419e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96f4105370 0x7f96f41a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f96f41138e0 con 0x7f96f41029c0 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96f4113760 con 0x7f96f4100050 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f8929640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96f4113a60 con 0x7f96f4105370 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f41029c0 0x7f96f419e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f41029c0 0x7f96f419e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55266/0 (socket says 172.21.15.78:55266) 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f1d9b640 1 -- 172.21.15.78:0/2556406280 learned_addr learned my addr 172.21.15.78:0/2556406280 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.082+0000 7f96f259c640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f4100050 0x7f96f419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f2d9d640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96f4105370 0x7f96f41a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f259c640 1 -- 172.21.15.78:0/2556406280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96f4105370 msgr2=0x7f96f41a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f259c640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96f4105370 0x7f96f41a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f259c640 1 -- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f41029c0 msgr2=0x7f96f419e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f259c640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f41029c0 0x7f96f419e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f259c640 1 -- 172.21.15.78:0/2556406280 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f41a5c10 con 0x7f96f4100050 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f259c640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f4100050 0x7f96f419dff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f96e400e9f0 tx=0x7f96e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96e37fe640 1 -- 172.21.15.78:0/2556406280 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e400ccc0 con 0x7f96f4100050 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f41a5f00 con 0x7f96f4100050 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96e37fe640 1 -- 172.21.15.78:0/2556406280 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96e40108e0 con 0x7f96f4100050 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.083+0000 7f96e37fe640 1 -- 172.21.15.78:0/2556406280 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4018880 con 0x7f96f4100050 2024-03-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.084+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f41a63c0 con 0x7f96f4100050 2024-03-16T11:07:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.085+0000 7f96e37fe640 1 -- 172.21.15.78:0/2556406280 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f96e4010a90 con 0x7f96f4100050 2024-03-16T11:07:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.086+0000 7f96e37fe640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96dc0a2800 0x7f96dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.086+0000 7f96e37fe640 1 -- 172.21.15.78:0/2556406280 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f96e40cf1f0 con 0x7f96f4100050 2024-03-16T11:07:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.086+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96dc0a2800 0x7f96dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.087+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96dc0a2800 0x7f96dc0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f96e80095f0 tx=0x7f96e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.087+0000 7f96fb5f2740 1 Processor -- start 2024-03-16T11:07:05.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96fb5f2740 1 -- start start 2024-03-16T11:07:05.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f20565ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f205662e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2056634f0 0x55f205665980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f2055c20d0 con 0x55f20565d990 2024-03-16T11:07:05.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f20556ac70 con 0x55f20565e380 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f2055be170 con 0x55f2056634f0 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f20565ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f20565ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55272/0 (socket says 172.21.15.78:55272) 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2056634f0 0x55f205665980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 -- 172.21.15.78:0/3959847378 learned_addr learned my addr 172.21.15.78:0/3959847378 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f205662e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 -- 172.21.15.78:0/3959847378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2056634f0 msgr2=0x55f205665980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2056634f0 0x55f205665980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 -- 172.21.15.78:0/3959847378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 msgr2=0x55f205662e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f205662e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 -- 172.21.15.78:0/3959847378 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f20539e7e0 con 0x55f20565d990 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f2d9d640 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2056634f0 0x55f205665980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.088+0000 7f96f259c640 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f20565ddb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f96e40a3e00 tx=0x7f96e409c290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7960e6292bcb893f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.089+0000 7f96e0ff9640 1 -- 172.21.15.78:0/3959847378 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e40a4330 con 0x55f20565d990 2024-03-16T11:07:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.089+0000 7f96e0ff9640 1 -- 172.21.15.78:0/3959847378 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96e4099e90 con 0x55f20565d990 2024-03-16T11:07:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.089+0000 7f96e0ff9640 1 -- 172.21.15.78:0/3959847378 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4097b40 con 0x55f20565d990 2024-03-16T11:07:05.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3959847378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 msgr2=0x55f20565ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f20565ddb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f96e40a3e00 tx=0x7f96e409c290 comp rx=0 tx=0).stop 2024-03-16T11:07:05.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3959847378 shutdown_connections 2024-03-16T11:07:05.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2056634f0 0x55f205665980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f205662e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3959847378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f20565ddb0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3959847378 >> 172.21.15.78:0/3959847378 conn(0x55f205625c50 msgr2=0x55f205659300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3959847378 shutdown_connections 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3959847378 wait complete. 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- start start 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 Processor -- start 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f205667e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565e380 0x55f205668350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f2056663e0 con 0x55f20565d990 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f20557e0d0 con 0x55f2056634f0 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f205666560 con 0x55f20565e380 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96f259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f205667e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96f259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f205667e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55274/0 (socket says 172.21.15.78:55274) 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f259c640 1 -- 172.21.15.78:0/1208534440 learned_addr learned my addr 172.21.15.78:0/1208534440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565e380 0x55f205668350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f2d9d640 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.090+0000 7f96fb5f2740 10 client.0 _ll_register_callbacks cb 0x55f205547760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f259c640 1 -- 172.21.15.78:0/1208534440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565e380 msgr2=0x55f205668350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f259c640 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565e380 0x55f205668350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f259c640 1 -- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 msgr2=0x55f20566f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f259c640 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f259c640 1 -- 172.21.15.78:0/1208534440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2053fef70 con 0x55f20565d990 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96f259c640 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f205667e10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f96e409bad0 tx=0x7f96e409fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e750fcb44ebce770 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96d9ffb640 1 -- 172.21.15.78:0/1208534440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e40a4330 con 0x55f20565d990 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96d9ffb640 1 -- 172.21.15.78:0/1208534440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f96e40ba270 con 0x55f20565d990 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.091+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e4099e90 con 0x55f20565d990 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96d9ffb640 1 -- 172.21.15.78:0/1208534440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f96e40ba470 con 0x55f20565d990 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 10 client.15729 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:05.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f205296170 con 0x55f20565d990 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96d9ffb640 1 -- 172.21.15.78:0/1208534440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96e4099e90 con 0x55f20565d990 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96d9ffb640 1 -- 172.21.15.78:0/1208534440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f96e40a0ce0 con 0x55f20565d990 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 10 client.15729 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 10 client.15729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 20 client.15729 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056758c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 4 client.15729 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2055cf610 con 0x55f2056734b0 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.092+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.093+0000 7f96d9ffb640 10 client.15729 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.093+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056758c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce788694dbb88a59 server_cookie=a1729113cb5d9cee in_seq=0 out_seq=0 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.093+0000 7f96d9ffb640 1 -- 172.21.15.78:0/1208534440 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2055cf610 con 0x55f2056734b0 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.093+0000 7f96d9ffb640 10 client.15729 handle_command_reply: tid=0 2024-03-16T11:07:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.093+0000 7f96fb5f2740 1 client.15729 shutdown 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.093+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 msgr2=0x55f205667e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.093+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f205667e10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f96e409bad0 tx=0x7f96e409fda0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 shutdown_connections 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056758c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96d9ffb640 0 client.15729 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565e380 0x55f205668350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/1208534440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565d990 0x55f205667e10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 >> 172.21.15.78:0/1208534440 conn(0x55f205625c50 msgr2=0x55f2056584e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 shutdown_connections 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 1 -- 172.21.15.78:0/1208534440 wait complete. 2024-03-16T11:07:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.094+0000 7f96fb5f2740 20 client.15729 trim_cache size 0 max 16384 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 Processor -- start 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 -- start start 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20565cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f20566f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2056634f0 0x55f205668130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f96f41b1af0 con 0x55f2056634f0 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96f41af3c0 con 0x55f20565e380 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96f41ad1f0 con 0x55f20565d990 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96f1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f20566f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96f259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20565cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2056634f0 0x55f205668130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96f259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20565cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54278/0 (socket says 172.21.15.78:54278) 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96f259c640 1 -- 172.21.15.78:0/3405724862 learned_addr learned my addr 172.21.15.78:0/3405724862 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.128+0000 7f96f1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f20566f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43286/0 (socket says 172.21.15.78:43286) 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96f1d9b640 1 -- 172.21.15.78:0/3405724862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 msgr2=0x55f20565cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20565cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96f1d9b640 1 -- 172.21.15.78:0/3405724862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2056634f0 msgr2=0x55f205668130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2056634f0 0x55f205668130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96f1d9b640 1 -- 172.21.15.78:0/3405724862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f205654d40 con 0x55f20565e380 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96f259c640 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20565cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f20566f320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55f205676170 tx=0x7f96e8014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d99b84b37c05587 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96dbfff640 1 -- 172.21.15.78:0/3405724862 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8031070 con 0x55f20565e380 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96dbfff640 1 -- 172.21.15.78:0/3405724862 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96e802c810 con 0x55f20565e380 2024-03-16T11:07:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.129+0000 7f96dbfff640 1 -- 172.21.15.78:0/3405724862 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8029da0 con 0x55f20565e380 2024-03-16T11:07:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3405724862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 msgr2=0x55f20566f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f20566f320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55f205676170 tx=0x7f96e8014000 comp rx=0 tx=0).stop 2024-03-16T11:07:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3405724862 shutdown_connections 2024-03-16T11:07:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2056634f0 0x55f205668130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f20565e380 0x55f20566f320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/3405724862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20565cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3405724862 >> 172.21.15.78:0/3405724862 conn(0x55f205625c50 msgr2=0x55f205667a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3405724862 shutdown_connections 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 -- 172.21.15.78:0/3405724862 wait complete. 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 -- start start 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 Processor -- start 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20566f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.130+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565e380 0x55f20566d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96fb5f2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f96f41ad880 con 0x55f20565e380 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96f41aecf0 con 0x55f2056634f0 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96fb5f2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96f41adf50 con 0x55f20565d990 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96fb5f2740 10 client.0 _ll_register_callbacks cb 0x7f96e8009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565e380 0x55f20566d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565e380 0x55f20566d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55288/0 (socket says 172.21.15.78:55288) 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 -- 172.21.15.78:0/2893705303 learned_addr learned my addr 172.21.15.78:0/2893705303 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f259c640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20566f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f2d9d640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 -- 172.21.15.78:0/2893705303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 msgr2=0x55f20566f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20566f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 -- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 msgr2=0x55f20566dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 -- 172.21.15.78:0/2893705303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f20566e6f0 con 0x55f20565e380 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f2d9d640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96f1d9b640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565e380 0x55f20566d940 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f96e801d490 tx=0x7f96e80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f67549d742b6fe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96d9ffb640 1 -- 172.21.15.78:0/2893705303 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8031050 con 0x55f20565e380 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96d9ffb640 1 -- 172.21.15.78:0/2893705303 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f96e8029ca0 con 0x55f20565e380 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.131+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e802c810 con 0x55f20565e380 2024-03-16T11:07:05.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96d9ffb640 1 -- 172.21.15.78:0/2893705303 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f96e802c160 con 0x55f20565e380 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 10 client.15735 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f2056da8e0 con 0x55f20565e380 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96d9ffb640 1 -- 172.21.15.78:0/2893705303 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96e8036450 con 0x55f20565e380 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96d9ffb640 1 -- 172.21.15.78:0/2893705303 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f96e8049400 con 0x55f20565e380 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 10 client.15735 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 20 client.15735 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056dd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 4 client.15735 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f96e802c810 con 0x55f2056734b0 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.132+0000 7f96f259c640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056dd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.133+0000 7f96d9ffb640 10 client.15735 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.133+0000 7f96f259c640 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056dd220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=404b088ffb133086 server_cookie=1562df8c46f9dbb4 in_seq=0 out_seq=0 2024-03-16T11:07:05.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96d9ffb640 1 -- 172.21.15.78:0/2893705303 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f96e802c810 con 0x55f2056734b0 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96d9ffb640 10 client.15735 handle_command_reply: tid=0 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 client.15735 shutdown 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565e380 msgr2=0x55f20566d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565e380 0x55f20566d940 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f96e801d490 tx=0x7f96e80056d0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 shutdown_connections 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f2056734b0 0x55f2056dd220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f2056634f0 0x55f20566dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96d9ffb640 0 client.15735 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f20565e380 0x55f20566d940 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 --2- 172.21.15.78:0/2893705303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f20565d990 0x55f20566f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 >> 172.21.15.78:0/2893705303 conn(0x55f205625c50 msgr2=0x55f205657c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 shutdown_connections 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 1 -- 172.21.15.78:0/2893705303 wait complete. 2024-03-16T11:07:05.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.134+0000 7f96fb5f2740 20 client.15735 trim_cache size 0 max 16384 2024-03-16T11:07:05.141 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:07:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-16T11:07:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "ae1f9c19-2bb5-40cd-abb9-9b59d3950073", 2024-03-16T11:07:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-16T11:07:05.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:07:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96dc0a2800 msgr2=0x7f96dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96dc0a2800 0x7f96dc0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f96e80095f0 tx=0x7f96e8005f90 comp rx=0 tx=0).stop 2024-03-16T11:07:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f4100050 msgr2=0x7f96f419dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f4100050 0x7f96f419dff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f96e400e9f0 tx=0x7f96e400eeb0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 shutdown_connections 2024-03-16T11:07:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f96dc0a2800 0x7f96dc0a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96f4105370 0x7f96f41a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96f41029c0 0x7f96f419e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 --2- 172.21.15.78:0/2556406280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96f4100050 0x7f96f419dff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 >> 172.21.15.78:0/2556406280 conn(0x7f96f40fbc70 msgr2=0x7f96f41009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 shutdown_connections 2024-03-16T11:07:05.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.135+0000 7f96f8929640 1 -- 172.21.15.78:0/2556406280 wait complete. 2024-03-16T11:07:05.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:07:05.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:07:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 -- 172.21.15.78:0/2576713609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c107520 msgr2=0x7f653c109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 --2- 172.21.15.78:0/2576713609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c107520 0x7f653c109910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6534009ca0 tx=0x7f653402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 -- 172.21.15.78:0/2576713609 shutdown_connections 2024-03-16T11:07:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 --2- 172.21.15.78:0/2576713609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c107520 0x7f653c109910 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 --2- 172.21.15.78:0/2576713609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f653c104bf0 0x7f653c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 --2- 172.21.15.78:0/2576713609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c1022c0 0x7f653c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 -- 172.21.15.78:0/2576713609 >> 172.21.15.78:0/2576713609 conn(0x7f653c0fbc70 msgr2=0x7f653c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 -- 172.21.15.78:0/2576713609 shutdown_connections 2024-03-16T11:07:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.252+0000 7f6540863640 1 -- 172.21.15.78:0/2576713609 wait complete. 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.253+0000 7f6540863640 1 Processor -- start 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.253+0000 7f6540863640 1 -- start start 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6540863640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c1022c0 0x7f653c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6540863640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c104bf0 0x7f653c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6540863640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f653c107520 0x7f653c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6540863640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f653c10d310 con 0x7f653c107520 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6540863640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f653c10d190 con 0x7f653c104bf0 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6540863640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f653c10d490 con 0x7f653c1022c0 2024-03-16T11:07:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c1022c0 0x7f653c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6539d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c104bf0 0x7f653c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f653c107520 0x7f653c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c1022c0 0x7f653c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54302/0 (socket says 172.21.15.78:54302) 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 -- 172.21.15.78:0/904333988 learned_addr learned my addr 172.21.15.78:0/904333988 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6539d9b640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c104bf0 0x7f653c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43310/0 (socket says 172.21.15.78:43310) 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 -- 172.21.15.78:0/904333988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c104bf0 msgr2=0x7f653c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c104bf0 0x7f653c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 -- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f653c107520 msgr2=0x7f653c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f653c107520 0x7f653c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f653a59c640 1 -- 172.21.15.78:0/904333988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c19f650 con 0x7f653c1022c0 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.254+0000 7f6539d9b640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c104bf0 0x7f653c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.255+0000 7f653a59c640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c1022c0 0x7f653c197a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e9f0 tx=0x7f652c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.255+0000 7f65337fe640 1 -- 172.21.15.78:0/904333988 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c00cd80 con 0x7f653c1022c0 2024-03-16T11:07:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.255+0000 7f65337fe640 1 -- 172.21.15.78:0/904333988 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f652c0108e0 con 0x7f653c1022c0 2024-03-16T11:07:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.255+0000 7f65337fe640 1 -- 172.21.15.78:0/904333988 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c018920 con 0x7f653c1022c0 2024-03-16T11:07:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.255+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c19f940 con 0x7f653c1022c0 2024-03-16T11:07:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.255+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c19fe40 con 0x7f653c1022c0 2024-03-16T11:07:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.257+0000 7f65337fe640 1 -- 172.21.15.78:0/904333988 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f652c010a90 con 0x7f653c1022c0 2024-03-16T11:07:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 Processor -- start 2024-03-16T11:07:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 -- start start 2024-03-16T11:07:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716104590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716109660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x55671610c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556716069500 con 0x556716109cd0 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556716011be0 con 0x556716104b60 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6543447740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5567160655a0 con 0x556716104170 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f653ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f653a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f653ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716104590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54318/0 (socket says 172.21.15.78:54318) 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f653ad9d640 1 -- 172.21.15.78:0/1512643050 learned_addr learned my addr 172.21.15.78:0/1512643050 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.258+0000 7f6539d9b640 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x55671610c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f65337fe640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f65280a2880 0x7f65280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653a59c640 1 -- 172.21.15.78:0/1512643050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 msgr2=0x556716104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653a59c640 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653a59c640 1 -- 172.21.15.78:0/1512643050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 msgr2=0x55671610c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653a59c640 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x55671610c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653a59c640 1 -- 172.21.15.78:0/1512643050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567160254b0 con 0x556716104b60 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653ad9d640 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653ad9d640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f65280a2880 0x7f65280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f65337fe640 1 -- 172.21.15.78:0/904333988 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f652c0ced00 con 0x7f653c1022c0 2024-03-16T11:07:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653a59c640 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716109660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f652c09b450 tx=0x7f652c0c60f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4274c6237726bb87 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f6530ff9640 1 -- 172.21.15.78:0/1512643050 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c026030 con 0x556716104b60 2024-03-16T11:07:05.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f6530ff9640 1 -- 172.21.15.78:0/1512643050 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f652c010de0 con 0x556716104b60 2024-03-16T11:07:05.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f6530ff9640 1 -- 172.21.15.78:0/1512643050 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c09b750 con 0x556716104b60 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.259+0000 7f653ad9d640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f65280a2880 0x7f65280a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6534009a60 tx=0x7f653402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 -- 172.21.15.78:0/1512643050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 msgr2=0x556716109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716109660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f652c09b450 tx=0x7f652c0c60f0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 -- 172.21.15.78:0/1512643050 shutdown_connections 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x55671610c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716109660 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 --2- 172.21.15.78:0/1512643050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 -- 172.21.15.78:0/1512643050 >> 172.21.15.78:0/1512643050 conn(0x5567160ff6e0 msgr2=0x5567160ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 -- 172.21.15.78:0/1512643050 shutdown_connections 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 -- 172.21.15.78:0/1512643050 wait complete. 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.260+0000 7f6543447740 1 -- start start 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 1 Processor -- start 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716112a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716112f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716119fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556716111030 con 0x556716109cd0 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556716025160 con 0x556716104b60 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5567161111b0 con 0x556716104170 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6543447740 10 client.0 _ll_register_callbacks cb 0x556716009150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6539d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716119fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716112a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6539d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716112f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43332/0 (socket says 172.21.15.78:43332) 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716112a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54322/0 (socket says 172.21.15.78:54322) 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6539d9b640 1 -- 172.21.15.78:0/3923985048 learned_addr learned my addr 172.21.15.78:0/3923985048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653ad9d640 1 -- 172.21.15.78:0/3923985048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 msgr2=0x556716112a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653ad9d640 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716112a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653ad9d640 1 -- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 msgr2=0x556716112f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653ad9d640 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716112f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653ad9d640 1 -- 172.21.15.78:0/3923985048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556715e477d0 con 0x556716109cd0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653a59c640 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716112a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6539d9b640 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716112f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f653ad9d640 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716119fb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6534032380 tx=0x7f65340324f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ba96bce312b1fc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.261+0000 7f6521ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6521ffb640 1 -- 172.21.15.78:0/3923985048 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6534052060 con 0x556716109cd0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6521ffb640 1 -- 172.21.15.78:0/3923985048 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f6534041dd0 con 0x556716109cd0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556716076a40 con 0x556716109cd0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6521ffb640 1 -- 172.21.15.78:0/3923985048 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6534046070 con 0x556716109cd0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 10 client.15747 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556715d7d3f0 con 0x556716109cd0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6521ffb640 1 -- 172.21.15.78:0/3923985048 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556716076a40 con 0x556716109cd0 2024-03-16T11:07:05.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6521ffb640 1 -- 172.21.15.78:0/3923985048 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6534057070 con 0x556716109cd0 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 10 client.15747 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 20 client.15747 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55671611e550 0x556716120980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 4 client.15747 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.262+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556716070d90 con 0x55671611e550 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.263+0000 7f653a59c640 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55671611e550 0x556716120980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.263+0000 7f6521ffb640 10 client.15747 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.263+0000 7f653a59c640 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55671611e550 0x556716120980 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=907cc9ec9f68c39d server_cookie=d36525d8dd9e650e in_seq=0 out_seq=0 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6521ffb640 1 -- 172.21.15.78:0/3923985048 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556716070d90 con 0x55671611e550 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6521ffb640 10 client.15747 handle_command_reply: tid=0 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 client.15747 shutdown 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 msgr2=0x556716119fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716119fb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6534032380 tx=0x7f65340324f0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 shutdown_connections 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55671611e550 0x556716120980 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716119fb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104b60 0x556716112f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 --2- 172.21.15.78:0/3923985048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104170 0x556716112a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6521ffb640 0 client.15747 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 >> 172.21.15.78:0/3923985048 conn(0x5567160ff6e0 msgr2=0x5567160fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 shutdown_connections 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 1 -- 172.21.15.78:0/3923985048 wait complete. 2024-03-16T11:07:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.264+0000 7f6543447740 20 client.15747 trim_cache size 0 max 16384 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 Processor -- start 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 -- start start 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x556716103050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716119f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716125240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1a9330 con 0x556716109cd0 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1a6c00 con 0x556716104170 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.281+0000 7f6543447740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1a4a30 con 0x556716104b60 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x556716103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f6539d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716119f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x556716103050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43340/0 (socket says 172.21.15.78:43340) 2024-03-16T11:07:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f6539d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716119f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:54326/0 (socket says 172.21.15.78:54326) 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 -- 172.21.15.78:0/2466284245 learned_addr learned my addr 172.21.15.78:0/2466284245 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716125240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 -- 172.21.15.78:0/2466284245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 msgr2=0x556716119f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 --2- 172.21.15.78:0/2466284245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716119f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 -- 172.21.15.78:0/2466284245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 msgr2=0x556716125240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 --2- 172.21.15.78:0/2466284245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716125240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 -- 172.21.15.78:0/2466284245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556715d3d0f0 con 0x556716104170 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f653a59c640 1 --2- 172.21.15.78:0/2466284245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x556716103050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f652c097bb0 tx=0x7f652c0cc6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b72c7c70b6923a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f6523fff640 1 -- 172.21.15.78:0/2466284245 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c09b750 con 0x556716104170 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.282+0000 7f6523fff640 1 -- 172.21.15.78:0/2466284245 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f652c0997f0 con 0x556716104170 2024-03-16T11:07:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6523fff640 1 -- 172.21.15.78:0/2466284245 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0cbc50 con 0x556716104170 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 -- 172.21.15.78:0/2466284245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 msgr2=0x556716103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 --2- 172.21.15.78:0/2466284245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x556716103050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f652c097bb0 tx=0x7f652c0cc6d0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 -- 172.21.15.78:0/2466284245 shutdown_connections 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 --2- 172.21.15.78:0/2466284245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716125240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 --2- 172.21.15.78:0/2466284245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716119f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 --2- 172.21.15.78:0/2466284245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x556716103050 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 -- 172.21.15.78:0/2466284245 >> 172.21.15.78:0/2466284245 conn(0x5567160ff6e0 msgr2=0x556716102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 -- 172.21.15.78:0/2466284245 shutdown_connections 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.283+0000 7f6543447740 1 -- 172.21.15.78:0/2466284245 wait complete. 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 -- start start 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 Processor -- start 2024-03-16T11:07:05.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x5567161177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716117ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716118ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1a50c0 con 0x556716109cd0 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1a6530 con 0x556716104170 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1a5790 con 0x556716104b60 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6543447740 10 client.0 _ll_register_callbacks cb 0x7f652c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x5567161177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716118ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f6539d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716117ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x5567161177a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:43348/0 (socket says 172.21.15.78:43348) 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716118ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55346/0 (socket says 172.21.15.78:55346) 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 -- 172.21.15.78:0/2124472836 learned_addr learned my addr 172.21.15.78:0/2124472836 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 -- 172.21.15.78:0/2124472836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 msgr2=0x556716117ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716117ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 -- 172.21.15.78:0/2124472836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 msgr2=0x556716118ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716118ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.284+0000 7f653a59c640 1 -- 172.21.15.78:0/2124472836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567160309d0 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f653a59c640 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x5567161177a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f652c0ce010 tx=0x7f652c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b40fe1c92d23e4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6521ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6521ffb640 1 -- 172.21.15.78:0/2124472836 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c09f7f0 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f652c0997f0 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6521ffb640 1 -- 172.21.15.78:0/2124472836 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f652c0cb8d0 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6521ffb640 1 -- 172.21.15.78:0/2124472836 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f652c0ad050 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 10 client.25586 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556716184880 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6521ffb640 1 -- 172.21.15.78:0/2124472836 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f652c0997f0 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6521ffb640 1 -- 172.21.15.78:0/2124472836 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f652c084ce0 con 0x556716104170 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 10 client.25586 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 10 client.25586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:05.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 20 client.25586 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5567161849d0 0x556716186dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 4 client.25586 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556715fee540 con 0x5567161849d0 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.285+0000 7f6539d9b640 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5567161849d0 0x556716186dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6521ffb640 10 client.25586 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6539d9b640 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5567161849d0 0x556716186dc0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74471a137ee743a7 server_cookie=579b75a5602512cd in_seq=0 out_seq=0 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6521ffb640 1 -- 172.21.15.78:0/2124472836 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556715fee540 con 0x5567161849d0 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6521ffb640 10 client.25586 handle_command_reply: tid=0 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 client.25586 shutdown 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 msgr2=0x5567161177a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x5567161177a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f652c0ce010 tx=0x7f652c000bc0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 shutdown_connections 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5567161849d0 0x556716186dc0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556716109cd0 0x556716118ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6521ffb640 0 client.25586 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556716104b60 0x556716117ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 --2- 172.21.15.78:0/2124472836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556716104170 0x5567161177a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 >> 172.21.15.78:0/2124472836 conn(0x5567160ff6e0 msgr2=0x5567161031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 shutdown_connections 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 1 -- 172.21.15.78:0/2124472836 wait complete. 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.286+0000 7f6543447740 20 client.25586 trim_cache size 0 max 16384 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ae1f9c19-2bb5-40cd-abb9-9b59d3950073": { 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-16T11:07:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "ae1f9c19-2bb5-40cd-abb9-9b59d3950073", 2024-03-16T11:07:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-16T11:07:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:07:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:07:05.293 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:07:05.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f65280a2880 msgr2=0x7f65280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f65280a2880 0x7f65280a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6534009a60 tx=0x7f653402e040 comp rx=0 tx=0).stop 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c1022c0 msgr2=0x7f653c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c1022c0 0x7f653c197a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e9f0 tx=0x7f652c00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 shutdown_connections 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f653c107520 0x7f653c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c104bf0 0x7f653c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f65280a2880 0x7f65280a4d40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 --2- 172.21.15.78:0/904333988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653c1022c0 0x7f653c197a80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 >> 172.21.15.78:0/904333988 conn(0x7f653c0fbc70 msgr2=0x7f653c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 shutdown_connections 2024-03-16T11:07:05.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:05.287+0000 7f6540863640 1 -- 172.21.15.78:0/904333988 wait complete. 2024-03-16T11:07:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:05 smithi078 ceph-mon[32235]: pgmap v381: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:07:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:05 smithi078 ceph-mon[32235]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-03-16T11:07:05.298 INFO:tasks.cephfs.filesystem:scrub status for tag:ae1f9c19-2bb5-40cd-abb9-9b59d3950073 - {'path': '/', 'tag': 'ae1f9c19-2bb5-40cd-abb9-9b59d3950073', 'options': 'recursive,force'} 2024-03-16T11:07:05.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:05 smithi098 ceph-mon[35848]: pgmap v381: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:07:05.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:05 smithi098 ceph-mon[35848]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-03-16T11:07:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:05 smithi153 ceph-mon[35521]: pgmap v381: 97 pgs: 97 active+clean; 662 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T11:07:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:05 smithi153 ceph-mon[35521]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-03-16T11:07:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 -- 172.21.15.78:0/3595118663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 msgr2=0x7fed441002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 --2- 172.21.15.78:0/3595118663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441002a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fed34009900 tx=0x7fed3402bd50 comp rx=0 tx=0).stop 2024-03-16T11:07:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 -- 172.21.15.78:0/3595118663 shutdown_connections 2024-03-16T11:07:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 --2- 172.21.15.78:0/3595118663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed44101990 0x7fed441082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 --2- 172.21.15.78:0/3595118663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed44100fd0 0x7fed44101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 --2- 172.21.15.78:0/3595118663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441002a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 -- 172.21.15.78:0/3595118663 >> 172.21.15.78:0/3595118663 conn(0x7fed440fb430 msgr2=0x7fed440fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 -- 172.21.15.78:0/3595118663 shutdown_connections 2024-03-16T11:07:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.811+0000 7fed49187640 1 -- 172.21.15.78:0/3595118663 wait complete. 2024-03-16T11:07:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 Processor -- start 2024-03-16T11:07:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 -- start start 2024-03-16T11:07:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed44100fd0 0x7fed44195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed44101990 0x7fed4419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fed4410b160 con 0x7fed44101990 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed4410afe0 con 0x7fed44100fd0 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.812+0000 7fed49187640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed4410b2e0 con 0x7fed440ffe80 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441957e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:46992/0 (socket says 172.21.15.78:46992) 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 -- 172.21.15.78:0/2995743515 learned_addr learned my addr 172.21.15.78:0/2995743515 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed4359e640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed44101990 0x7fed4419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed4259c640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed44100fd0 0x7fed44195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 -- 172.21.15.78:0/2995743515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed44100fd0 msgr2=0x7fed44195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed44100fd0 0x7fed44195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 -- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed44101990 msgr2=0x7fed4419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed44101990 0x7fed4419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 -- 172.21.15.78:0/2995743515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4419d400 con 0x7fed440ffe80 2024-03-16T11:07:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed4259c640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed44100fd0 0x7fed44195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed42d9d640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441957e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fed34006010 tx=0x7fed34005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed2bfff640 1 -- 172.21.15.78:0/2995743515 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed34042020 con 0x7fed440ffe80 2024-03-16T11:07:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed2bfff640 1 -- 172.21.15.78:0/2995743515 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed340052b0 con 0x7fed440ffe80 2024-03-16T11:07:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4419d690 con 0x7fed440ffe80 2024-03-16T11:07:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.813+0000 7fed2bfff640 1 -- 172.21.15.78:0/2995743515 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed34003db0 con 0x7fed440ffe80 2024-03-16T11:07:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.814+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4419db90 con 0x7fed440ffe80 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 Processor -- start 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 -- start start 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f539560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539b30 0x55638f53e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f540ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55638f495a40 con 0x55638f53e730 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55638f43e920 con 0x55638f539b30 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed4bd73740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55638f491ae0 con 0x55638f539140 2024-03-16T11:07:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.816+0000 7fed2bfff640 1 -- 172.21.15.78:0/2995743515 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fed34045020 con 0x7fed440ffe80 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f539560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f539560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:47008/0 (socket says 172.21.15.78:47008) 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 -- 172.21.15.78:0/2408278066 learned_addr learned my addr 172.21.15.78:0/2408278066 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4259c640 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f540ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed42d9d640 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539b30 0x55638f53e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 -- 172.21.15.78:0/2408278066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539b30 msgr2=0x55638f53e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539b30 0x55638f53e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 -- 172.21.15.78:0/2408278066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 msgr2=0x55638f540ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f540ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 -- 172.21.15.78:0/2408278066 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638f456460 con 0x55638f539140 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed2bfff640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fed2c0a28a0 0x7fed2c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4259c640 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f540ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed42d9d640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fed2c0a28a0 0x7fed2c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed4359e640 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f539560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009ef0 tx=0x7fed3c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85c79c39d0a6e9e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed2bfff640 1 -- 172.21.15.78:0/2995743515 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fed340f0bf0 con 0x7fed440ffe80 2024-03-16T11:07:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.817+0000 7fed297fa640 1 -- 172.21.15.78:0/2408278066 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c01a030 con 0x55638f539140 2024-03-16T11:07:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed297fa640 1 -- 172.21.15.78:0/2408278066 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed3c003970 con 0x55638f539140 2024-03-16T11:07:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed297fa640 1 -- 172.21.15.78:0/2408278066 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c004560 con 0x55638f539140 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 -- 172.21.15.78:0/2408278066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 msgr2=0x55638f539560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f539560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009ef0 tx=0x7fed3c009500 comp rx=0 tx=0).stop 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed42d9d640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fed2c0a28a0 0x7fed2c0a4d60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fed340b99b0 tx=0x7fed34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 -- 172.21.15.78:0/2408278066 shutdown_connections 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f540ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539b30 0x55638f53e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 --2- 172.21.15.78:0/2408278066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f539560 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 -- 172.21.15.78:0/2408278066 >> 172.21.15.78:0/2408278066 conn(0x55638f52b830 msgr2=0x55638f52bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 -- 172.21.15.78:0/2408278066 shutdown_connections 2024-03-16T11:07:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.818+0000 7fed4bd73740 1 -- 172.21.15.78:0/2408278066 wait complete. 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 -- start start 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 Processor -- start 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539140 0x55638f542ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f543410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f54a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55638f5416c0 con 0x55638f539140 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55638f451d70 con 0x55638f53e730 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55638f541840 con 0x55638f539b30 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4bd73740 10 client.0 _ll_register_callbacks cb 0x55638f1a9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539140 0x55638f542ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539140 0x55638f542ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55160/0 (socket says 172.21.15.78:55160) 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f543410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 -- 172.21.15.78:0/3567145713 learned_addr learned my addr 172.21.15.78:0/3567145713 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f54a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 -- 172.21.15.78:0/3567145713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 msgr2=0x55638f543410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f543410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 -- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 msgr2=0x55638f54a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f54a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.819+0000 7fed4359e640 1 -- 172.21.15.78:0/3567145713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638f1aa210 con 0x55638f539140 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed4359e640 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539140 0x55638f542ed0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009ec0 tx=0x7fed3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b6d5d86e485060 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed26ff5640 1 -- 172.21.15.78:0/3567145713 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c004720 con 0x55638f539140 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed26ff5640 1 -- 172.21.15.78:0/3567145713 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fed3c005000 con 0x55638f539140 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed3c003970 con 0x55638f539140 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed26ff5640 1 -- 172.21.15.78:0/3567145713 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fed3c005200 con 0x55638f539140 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed4bd73740 10 client.15765 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55638f276620 con 0x55638f539140 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed26ff5640 1 -- 172.21.15.78:0/3567145713 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed3c003970 con 0x55638f539140 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed26ff5640 1 -- 172.21.15.78:0/3567145713 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fed3c01e6c0 con 0x55638f539140 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.820+0000 7fed4bd73740 10 client.15765 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed4bd73740 10 client.15765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed4bd73740 20 client.15765 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f54ea00 0x55638f550df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed4bd73740 4 client.15765 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55638f4a2f80 con 0x55638f54ea00 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed4259c640 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f54ea00 0x55638f550df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed26ff5640 10 client.15765 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.821+0000 7fed4259c640 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f54ea00 0x55638f550df0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b72d77fb378d2a49 server_cookie=62c213c65c39b6c7 in_seq=0 out_seq=0 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed26ff5640 1 -- 172.21.15.78:0/3567145713 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55638f4a2f80 con 0x55638f54ea00 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed26ff5640 10 client.15765 handle_command_reply: tid=0 2024-03-16T11:07:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 client.15765 shutdown 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539140 msgr2=0x55638f542ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539140 0x55638f542ed0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009ec0 tx=0x7fed3c009c00 comp rx=0 tx=0).stop 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 shutdown_connections 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f54ea00 0x55638f550df0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f54a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f543410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed26ff5640 0 client.15765 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3567145713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539140 0x55638f542ed0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 >> 172.21.15.78:0/3567145713 conn(0x55638f52b830 msgr2=0x55638f5387d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 shutdown_connections 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 1 -- 172.21.15.78:0/3567145713 wait complete. 2024-03-16T11:07:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.822+0000 7fed4bd73740 20 client.15765 trim_cache size 0 max 16384 2024-03-16T11:07:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 Processor -- start 2024-03-16T11:07:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 -- start start 2024-03-16T11:07:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539140 0x55638f543370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f54a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f52a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fed441a9330 con 0x55638f53e730 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed441a6c00 con 0x55638f539140 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4bd73740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed441a4a30 con 0x55638f539b30 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f52a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f54a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f52a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55168/0 (socket says 172.21.15.78:55168) 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4359e640 1 -- 172.21.15.78:0/3218561037 learned_addr learned my addr 172.21.15.78:0/3218561037 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f54a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:47028/0 (socket says 172.21.15.78:47028) 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.837+0000 7fed4359e640 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539140 0x55638f543370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed4259c640 1 -- 172.21.15.78:0/3218561037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539140 msgr2=0x55638f543370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed4259c640 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539140 0x55638f543370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed4259c640 1 -- 172.21.15.78:0/3218561037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 msgr2=0x55638f52a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed4259c640 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f52a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed4259c640 1 -- 172.21.15.78:0/3218561037 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638f51d140 con 0x55638f539b30 2024-03-16T11:07:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed4259c640 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f54a410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55638f5515b0 tx=0x7fed38005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=813cdd4280d34cd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed28ff9640 1 -- 172.21.15.78:0/3218561037 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38005020 con 0x55638f539b30 2024-03-16T11:07:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed28ff9640 1 -- 172.21.15.78:0/3218561037 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed380201b0 con 0x55638f539b30 2024-03-16T11:07:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.838+0000 7fed28ff9640 1 -- 172.21.15.78:0/3218561037 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38004a00 con 0x55638f539b30 2024-03-16T11:07:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 -- 172.21.15.78:0/3218561037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 msgr2=0x55638f54a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f54a410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55638f5515b0 tx=0x7fed38005f90 comp rx=0 tx=0).stop 2024-03-16T11:07:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 -- 172.21.15.78:0/3218561037 shutdown_connections 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f53e730 0x55638f52a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539b30 0x55638f54a410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 --2- 172.21.15.78:0/3218561037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f539140 0x55638f543370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 -- 172.21.15.78:0/3218561037 >> 172.21.15.78:0/3218561037 conn(0x55638f52b830 msgr2=0x55638f52bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 -- 172.21.15.78:0/3218561037 shutdown_connections 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 -- 172.21.15.78:0/3218561037 wait complete. 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 -- start start 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 Processor -- start 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f547b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.839+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539b30 0x55638f5480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4bd73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f549080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4bd73740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fed441a50c0 con 0x55638f539b30 2024-03-16T11:07:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4bd73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed441a6530 con 0x55638f53e730 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4bd73740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed441a5790 con 0x55638f539140 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4bd73740 10 client.0 _ll_register_callbacks cb 0x7fed38009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f547b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f547b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:47032/0 (socket says 172.21.15.78:47032) 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 -- 172.21.15.78:0/643174359 learned_addr learned my addr 172.21.15.78:0/643174359 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f549080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4259c640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539b30 0x55638f5480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 -- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 msgr2=0x55638f549080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f549080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 -- 172.21.15.78:0/643174359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539b30 msgr2=0x55638f5480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539b30 0x55638f5480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 -- 172.21.15.78:0/643174359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638f549670 con 0x55638f539140 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4259c640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539b30 0x55638f5480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed4359e640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f547b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fed3c003f50 tx=0x7fed3c004d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6679e60a001e146 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.840+0000 7fed26ff5640 1 -- 172.21.15.78:0/643174359 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c01a070 con 0x55638f539140 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed26ff5640 1 -- 172.21.15.78:0/643174359 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fed3c003970 con 0x55638f539140 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed380201b0 con 0x55638f539140 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed26ff5640 1 -- 172.21.15.78:0/643174359 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fed3c003b70 con 0x55638f539140 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 10 client.25591 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55638f5b4ca0 con 0x55638f539140 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed26ff5640 1 -- 172.21.15.78:0/643174359 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed3c01e8b0 con 0x55638f539140 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed26ff5640 1 -- 172.21.15.78:0/643174359 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fed3c005780 con 0x55638f539140 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 10 client.25591 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 20 client.25591 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f5b55e0 0x55638f5b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 4 client.25591 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:07:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fed380201b0 con 0x55638f5b55e0 2024-03-16T11:07:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.841+0000 7fed4259c640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f5b55e0 0x55638f5b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.842+0000 7fed26ff5640 10 client.25591 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.842+0000 7fed4259c640 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f5b55e0 0x55638f5b7a00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfeab6d1956920c1 server_cookie=7ac0409c5ca488e5 in_seq=0 out_seq=0 2024-03-16T11:07:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed26ff5640 1 -- 172.21.15.78:0/643174359 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fed380201b0 con 0x55638f5b55e0 2024-03-16T11:07:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed26ff5640 10 client.25591 handle_command_reply: tid=0 2024-03-16T11:07:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 client.25591 shutdown 2024-03-16T11:07:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 msgr2=0x55638f547b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f547b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fed3c003f50 tx=0x7fed3c004d50 comp rx=0 tx=0).stop 2024-03-16T11:07:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 shutdown_connections 2024-03-16T11:07:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55638f5b55e0 0x55638f5b7a00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638f53e730 0x55638f549080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55638f539b30 0x55638f5480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 --2- 172.21.15.78:0/643174359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55638f539140 0x55638f547b80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 >> 172.21.15.78:0/643174359 conn(0x55638f52b830 msgr2=0x55638f53db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed26ff5640 0 client.25591 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 shutdown_connections 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 1 -- 172.21.15.78:0/643174359 wait complete. 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.847+0000 7fed4bd73740 20 client.25591 trim_cache size 0 max 16384 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:07:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:07:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fed2c0a28a0 msgr2=0x7fed2c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fed2c0a28a0 0x7fed2c0a4d60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fed340b99b0 tx=0x7fed34000bc0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 msgr2=0x7fed441957e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441957e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fed34006010 tx=0x7fed34005e40 comp rx=0 tx=0).stop 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 shutdown_connections 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed44101990 0x7fed4419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed44100fd0 0x7fed44195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fed2c0a28a0 0x7fed2c0a4d60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 --2- 172.21.15.78:0/2995743515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed440ffe80 0x7fed441957e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 >> 172.21.15.78:0/2995743515 conn(0x7fed440fb430 msgr2=0x7fed44104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 shutdown_connections 2024-03-16T11:07:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:05.848+0000 7fed49187640 1 -- 172.21.15.78:0/2995743515 wait complete. 2024-03-16T11:07:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:06 smithi098 ceph-mon[35848]: scrub queued for path: / 2024-03-16T11:07:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:06 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:06 smithi098 ceph-mon[35848]: scrub summary: active paths [/] 2024-03-16T11:07:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:06 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:06.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:06 smithi153 ceph-mon[35521]: scrub queued for path: / 2024-03-16T11:07:06.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:06 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:06.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:06 smithi153 ceph-mon[35521]: scrub summary: active paths [/] 2024-03-16T11:07:06.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:06 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:06.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:06 smithi078 ceph-mon[32235]: scrub queued for path: / 2024-03-16T11:07:06.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:06 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:06.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:06 smithi078 ceph-mon[32235]: scrub summary: active paths [/] 2024-03-16T11:07:06.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:06 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:07.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:07 smithi078 ceph-mon[32235]: pgmap v382: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-16T11:07:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:07 smithi098 ceph-mon[35848]: pgmap v382: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-16T11:07:07.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:07 smithi153 ceph-mon[35521]: pgmap v382: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-16T11:07:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:09 smithi098 ceph-mon[35848]: pgmap v383: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T11:07:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:09 smithi153 ceph-mon[35521]: pgmap v383: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T11:07:09.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:09 smithi078 ceph-mon[32235]: pgmap v383: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T11:07:10.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:10 smithi098 ceph-mon[35848]: scrub summary: idle 2024-03-16T11:07:10.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:10 smithi153 ceph-mon[35521]: scrub summary: idle 2024-03-16T11:07:10.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:10 smithi078 ceph-mon[32235]: scrub summary: idle 2024-03-16T11:07:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:11 smithi098 ceph-mon[35848]: pgmap v384: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T11:07:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:11 smithi153 ceph-mon[35521]: pgmap v384: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T11:07:11.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:11 smithi078 ceph-mon[32235]: pgmap v384: 97 pgs: 97 active+clean; 663 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T11:07:13.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:13 smithi098 ceph-mon[35848]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T11:07:13.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:13 smithi153 ceph-mon[35521]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T11:07:13.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:13 smithi078 ceph-mon[32235]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T11:07:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:15 smithi098 ceph-mon[35848]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-16T11:07:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:15 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:07:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:15 smithi153 ceph-mon[35521]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-16T11:07:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:15 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:07:15.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:15 smithi078 ceph-mon[32235]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-16T11:07:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:15 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:07:16.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:16.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:16.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:16 smithi078 ceph-mon[32235]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-16T11:07:16.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:07:16.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:07:16.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:16 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:16 smithi098 ceph-mon[35848]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-16T11:07:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:07:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:07:16.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:16 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:16 smithi153 ceph-mon[35521]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-16T11:07:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:07:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:07:17.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:16 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/1078605129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d281074e0 msgr2=0x7f1d28109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/1078605129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d281074e0 0x7f1d28109930 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009ca0 tx=0x7f1d2002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/1078605129 shutdown_connections 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/1078605129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d281074e0 0x7f1d28109930 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/1078605129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d28104bb0 0x7f1d28106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/1078605129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d28102280 0x7f1d28104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/1078605129 >> 172.21.15.78:0/1078605129 conn(0x7f1d280fbc70 msgr2=0x7f1d280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/1078605129 shutdown_connections 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/1078605129 wait complete. 2024-03-16T11:07:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 Processor -- start 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 -- start start 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d28102280 0x7f1d28197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.938+0000 7f1d2e8c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d28104bb0 0x7f1d28197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d2e8c6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d281074e0 0x7f1d2819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d2e8c6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2810d330 con 0x7f1d28102280 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d2e8c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2810d1b0 con 0x7f1d28104bb0 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d2e8c6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2810d4b0 con 0x7f1d281074e0 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d28102280 0x7f1d28197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d28102280 0x7f1d28197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57438/0 (socket says 172.21.15.78:57438) 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 -- 172.21.15.78:0/3142213445 learned_addr learned my addr 172.21.15.78:0/3142213445 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d277fe640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d28104bb0 0x7f1d28197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d2ce63640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d281074e0 0x7f1d2819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 -- 172.21.15.78:0/3142213445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d281074e0 msgr2=0x7f1d2819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d281074e0 0x7f1d2819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 -- 172.21.15.78:0/3142213445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d28104bb0 msgr2=0x7f1d28197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d28104bb0 0x7f1d28197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 -- 172.21.15.78:0/3142213445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2819f590 con 0x7f1d28102280 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d27fff640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d28102280 0x7f1d28197970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800e9f0 tx=0x7f1d1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d257fa640 1 -- 172.21.15.78:0/3142213445 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1800cd80 con 0x7f1d28102280 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2819f880 con 0x7f1d28102280 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d257fa640 1 -- 172.21.15.78:0/3142213445 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d180108e0 con 0x7f1d28102280 2024-03-16T11:07:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d257fa640 1 -- 172.21.15.78:0/3142213445 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18018920 con 0x7f1d28102280 2024-03-16T11:07:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.939+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2819fe20 con 0x7f1d28102280 2024-03-16T11:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.941+0000 7f1d257fa640 1 -- 172.21.15.78:0/3142213445 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1d18010a90 con 0x7f1d28102280 2024-03-16T11:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 Processor -- start 2024-03-16T11:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 -- start start 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb3cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb41c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb44740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55843ea9da40 con 0x55843eb3c750 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55843ea46920 con 0x55843eb3d140 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d3158d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55843ea99ae0 con 0x55843eb422b0 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d257fa640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1d100a28a0 0x7f1d100a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d2ce63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb3cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb41c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d2ce63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb3cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57440/0 (socket says 172.21.15.78:57440) 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d2ce63640 1 -- 172.21.15.78:0/3071292852 learned_addr learned my addr 172.21.15.78:0/3071292852 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d277fe640 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb44740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d2ce63640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1d100a28a0 0x7f1d100a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d257fa640 1 -- 172.21.15.78:0/3142213445 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1d180ced70 con 0x7f1d28102280 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d27fff640 1 -- 172.21.15.78:0/3071292852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 msgr2=0x55843eb44740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d27fff640 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb44740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d27fff640 1 -- 172.21.15.78:0/3071292852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 msgr2=0x55843eb3cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d27fff640 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb3cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d27fff640 1 -- 172.21.15.78:0/3071292852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55843ea5e460 con 0x55843eb3d140 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d277fe640 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb44740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.942+0000 7f1d2ce63640 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb3cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.943+0000 7f1d27fff640 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb41c40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d18098190 tx=0x7f1d1809b120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37957d3e104daa3c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.943+0000 7f1d2ce63640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1d100a28a0 0x7f1d100a4d60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55843eb3d9a0 tx=0x7f1d2002c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.943+0000 7f1d0e7fc640 1 -- 172.21.15.78:0/3071292852 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18010de0 con 0x55843eb3d140 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.943+0000 7f1d0e7fc640 1 -- 172.21.15.78:0/3071292852 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d1809f860 con 0x55843eb3d140 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.943+0000 7f1d0e7fc640 1 -- 172.21.15.78:0/3071292852 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18097e30 con 0x55843eb3d140 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- 172.21.15.78:0/3071292852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 msgr2=0x55843eb41c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb41c40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d18098190 tx=0x7f1d1809b120 comp rx=0 tx=0).stop 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- 172.21.15.78:0/3071292852 shutdown_connections 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb44740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb41c40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 --2- 172.21.15.78:0/3071292852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb3cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- 172.21.15.78:0/3071292852 >> 172.21.15.78:0/3071292852 conn(0x55843eb338b0 msgr2=0x55843eb33cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- 172.21.15.78:0/3071292852 shutdown_connections 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- 172.21.15.78:0/3071292852 wait complete. 2024-03-16T11:07:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- start start 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 Processor -- start 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3c750 0x55843eb46a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb46fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb422b0 0x55843eb4df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55843eb45200 con 0x55843eb422b0 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55843ea59d70 con 0x55843eb3d140 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55843eb45380 con 0x55843eb3c750 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.944+0000 7f1d3158d740 10 client.0 _ll_register_callbacks cb 0x55843e7b1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d2ce63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb422b0 0x55843eb4df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb46fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb46fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:44958/0 (socket says 172.21.15.78:44958) 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d2ce63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb422b0 0x55843eb4df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57448/0 (socket says 172.21.15.78:57448) 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d277fe640 1 -- 172.21.15.78:0/4082532515 learned_addr learned my addr 172.21.15.78:0/4082532515 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d27fff640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3c750 0x55843eb46a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d27fff640 1 -- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 msgr2=0x55843eb46fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d27fff640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb46fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d27fff640 1 -- 172.21.15.78:0/4082532515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb422b0 msgr2=0x55843eb4df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d27fff640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb422b0 0x55843eb4df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d27fff640 1 -- 172.21.15.78:0/4082532515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55843e7b2210 con 0x55843eb3c750 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d277fe640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb46fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d2ce63640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb422b0 0x55843eb4df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d27fff640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3c750 0x55843eb46a80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55843eb44eb0 tx=0x7f1d1809b250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9172c9a1c896355 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d077fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d077fe640 1 -- 172.21.15.78:0/4082532515 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d180b3030 con 0x55843eb3c750 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d077fe640 1 -- 172.21.15.78:0/4082532515 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1d1809bec0 con 0x55843eb3c750 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d1809f860 con 0x55843eb3c750 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d077fe640 1 -- 172.21.15.78:0/4082532515 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1d1809c0c0 con 0x55843eb3c750 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d3158d740 10 client.25603 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.945+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55843e87e620 con 0x55843eb3c750 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d077fe640 1 -- 172.21.15.78:0/4082532515 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d1809f860 con 0x55843eb3c750 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d077fe640 1 -- 172.21.15.78:0/4082532515 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1d1809ad30 con 0x55843eb3c750 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d3158d740 10 client.25603 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d3158d740 10 client.25603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d3158d740 20 client.25603 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d3158d740 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843eb52480 0x55843eb54870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d3158d740 4 client.25603 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55843eaaaf80 con 0x55843eb52480 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d277fe640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843eb52480 0x55843eb54870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d077fe640 10 client.25603 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.946+0000 7f1d277fe640 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843eb52480 0x55843eb54870 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cf706ade7d3a9b8 server_cookie=a6caefd88f67d65c in_seq=0 out_seq=0 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.947+0000 7f1d077fe640 1 -- 172.21.15.78:0/4082532515 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55843eaaaf80 con 0x55843eb52480 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.947+0000 7f1d077fe640 10 client.25603 handle_command_reply: tid=0 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.947+0000 7f1d3158d740 1 client.25603 shutdown 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3c750 msgr2=0x55843eb46a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3c750 0x55843eb46a80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55843eb44eb0 tx=0x7f1d1809b250 comp rx=0 tx=0).stop 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 shutdown_connections 2024-03-16T11:07:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843eb52480 0x55843eb54870 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb422b0 0x55843eb4df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3d140 0x55843eb46fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 --2- 172.21.15.78:0/4082532515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3c750 0x55843eb46a80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d077fe640 0 client.25603 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 >> 172.21.15.78:0/4082532515 conn(0x55843eb338b0 msgr2=0x55843eb32bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 shutdown_connections 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 1 -- 172.21.15.78:0/4082532515 wait complete. 2024-03-16T11:07:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.948+0000 7f1d3158d740 20 client.25603 trim_cache size 0 max 16384 2024-03-16T11:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.962+0000 7f1d3158d740 1 Processor -- start 2024-03-16T11:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.962+0000 7f1d3158d740 1 -- start start 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.962+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3c750 0x55843eb46690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.962+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3d140 0x55843eb4df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.962+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb46bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.962+0000 7f1d3158d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281a9390 con 0x55843eb3d140 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.962+0000 7f1d3158d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281a6c60 con 0x55843eb3c750 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d3158d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281a4a90 con 0x55843eb422b0 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3d140 0x55843eb4df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3c750 0x55843eb46690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3c750 0x55843eb46690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:44972/0 (socket says 172.21.15.78:44972) 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3d140 0x55843eb4df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57462/0 (socket says 172.21.15.78:57462) 2024-03-16T11:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d27fff640 1 -- 172.21.15.78:0/742713745 learned_addr learned my addr 172.21.15.78:0/742713745 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d2ce63640 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb46bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 -- 172.21.15.78:0/742713745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 msgr2=0x55843eb46bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb46bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 -- 172.21.15.78:0/742713745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3c750 msgr2=0x55843eb46690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3c750 0x55843eb46690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 -- 172.21.15.78:0/742713745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55843eb25140 con 0x55843eb3d140 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d27fff640 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3c750 0x55843eb46690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d277fe640 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3d140 0x55843eb4df50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x55843eb54e40 tx=0x7f1d1c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a0b09d1cd3b924 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d0dffb640 1 -- 172.21.15.78:0/742713745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c005220 con 0x55843eb3d140 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d0dffb640 1 -- 172.21.15.78:0/742713745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d1c004c50 con 0x55843eb3d140 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.963+0000 7f1d0dffb640 1 -- 172.21.15.78:0/742713745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0205d0 con 0x55843eb3d140 2024-03-16T11:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 -- 172.21.15.78:0/742713745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3d140 msgr2=0x55843eb4df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3d140 0x55843eb4df50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x55843eb54e40 tx=0x7f1d1c005f90 comp rx=0 tx=0).stop 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 -- 172.21.15.78:0/742713745 shutdown_connections 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb422b0 0x55843eb46bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3d140 0x55843eb4df50 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 --2- 172.21.15.78:0/742713745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb3c750 0x55843eb46690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 -- 172.21.15.78:0/742713745 >> 172.21.15.78:0/742713745 conn(0x55843eb338b0 msgr2=0x55843eb2ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 -- 172.21.15.78:0/742713745 shutdown_connections 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 -- 172.21.15.78:0/742713745 wait complete. 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.964+0000 7f1d3158d740 1 -- start start 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 1 Processor -- start 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb4b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3d140 0x55843eb4b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb422b0 0x55843eb4c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281a5120 con 0x55843eb3c750 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281a6590 con 0x55843eb422b0 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281a57f0 con 0x55843eb3d140 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d3158d740 10 client.0 _ll_register_callbacks cb 0x7f1d1c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb4b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb4b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57470/0 (socket says 172.21.15.78:57470) 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 -- 172.21.15.78:0/1646122593 learned_addr learned my addr 172.21.15.78:0/1646122593 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d2ce63640 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb422b0 0x55843eb4c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d277fe640 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3d140 0x55843eb4b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 -- 172.21.15.78:0/1646122593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3d140 msgr2=0x55843eb4b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3d140 0x55843eb4b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 -- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb422b0 msgr2=0x55843eb4c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb422b0 0x55843eb4c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 -- 172.21.15.78:0/1646122593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55843eb4ccd0 con 0x55843eb3c750 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d27fff640 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb4b200 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1d18098140 tx=0x7f1d180b45b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16207a6104035b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.965+0000 7f1d077fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d077fe640 1 -- 172.21.15.78:0/1646122593 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d180b3040 con 0x55843eb3c750 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d077fe640 1 -- 172.21.15.78:0/1646122593 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1d1809c630 con 0x55843eb3c750 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d1c004c50 con 0x55843eb3c750 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d077fe640 1 -- 172.21.15.78:0/1646122593 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1d1809c830 con 0x55843eb3c750 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 10 client.15789 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55843eb4ccd0 con 0x55843eb3c750 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d077fe640 1 -- 172.21.15.78:0/1646122593 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d1c004c50 con 0x55843eb3c750 2024-03-16T11:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d077fe640 1 -- 172.21.15.78:0/1646122593 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1d180afce0 con 0x55843eb3c750 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 10 client.15789 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 20 client.15789 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843ebb86d0 0x55843ebbaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 4 client.15789 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55843eaa52d0 con 0x55843ebb86d0 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.966+0000 7f1d277fe640 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843ebb86d0 0x55843ebbaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.967+0000 7f1d077fe640 10 client.15789 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.967+0000 7f1d277fe640 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843ebb86d0 0x55843ebbaac0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b5400d662b7243 server_cookie=9fd48328728e12b2 in_seq=0 out_seq=0 2024-03-16T11:07:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.969+0000 7f1d077fe640 1 -- 172.21.15.78:0/1646122593 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55843eaa52d0 con 0x55843ebb86d0 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.969+0000 7f1d077fe640 10 client.15789 handle_command_reply: tid=0 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.969+0000 7f1d3158d740 1 client.15789 shutdown 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 msgr2=0x55843eb4b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb4b200 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1d18098140 tx=0x7f1d180b45b0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 shutdown_connections 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55843ebb86d0 0x55843ebbaac0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843eb422b0 0x55843eb4c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55843eb3d140 0x55843eb4b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d077fe640 0 client.15789 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 --2- 172.21.15.78:0/1646122593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55843eb3c750 0x55843eb4b200 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 >> 172.21.15.78:0/1646122593 conn(0x55843eb338b0 msgr2=0x55843eb46690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 shutdown_connections 2024-03-16T11:07:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 1 -- 172.21.15.78:0/1646122593 wait complete. 2024-03-16T11:07:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d3158d740 20 client.15789 trim_cache size 0 max 16384 2024-03-16T11:07:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:07:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:07:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:07:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:07:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1d100a28a0 msgr2=0x7f1d100a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1d100a28a0 0x7f1d100a4d60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55843eb3d9a0 tx=0x7f1d2002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d28102280 msgr2=0x7f1d28197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d28102280 0x7f1d28197970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800e9f0 tx=0x7f1d1800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.970+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 shutdown_connections 2024-03-16T11:07:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.971+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1d100a28a0 0x7f1d100a4d60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.971+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d281074e0 0x7f1d2819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.971+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d28104bb0 0x7f1d28197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.971+0000 7f1d2e8c6640 1 --2- 172.21.15.78:0/3142213445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d28102280 0x7f1d28197970 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.971+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 >> 172.21.15.78:0/3142213445 conn(0x7f1d280fbc70 msgr2=0x7f1d28102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.971+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 shutdown_connections 2024-03-16T11:07:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:18.971+0000 7f1d2e8c6640 1 -- 172.21.15.78:0/3142213445 wait complete. 2024-03-16T11:07:19.369 INFO:tasks.workunit.client.0.smithi078.stderr:Note: switching to '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'. 2024-03-16T11:07:19.369 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:19.369 INFO:tasks.workunit.client.0.smithi078.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-16T11:07:19.369 INFO:tasks.workunit.client.0.smithi078.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-16T11:07:19.369 INFO:tasks.workunit.client.0.smithi078.stderr:state without impacting any branches by switching back to a branch. 2024-03-16T11:07:19.369 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr: git switch -c 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr:Or undo this operation with: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr: git switch - 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:19.370 INFO:tasks.workunit.client.0.smithi078.stderr:HEAD is now at 3d78f5ba07b Merge PR #54581 into wip-batrick-testing-20240314.183243 2024-03-16T11:07:19.374 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-16T11:07:19.441 INFO:tasks.workunit.client.0.smithi078.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-16T11:07:19.443 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T11:07:19.443 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-16T11:07:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:19 smithi098 ceph-mon[35848]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:07:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:19 smithi153 ceph-mon[35521]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:07:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:19 smithi078 ceph-mon[32235]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T11:07:20.179 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-16T11:07:20.238 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-16T11:07:20.282 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T11:07:20.282 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T11:07:20.282 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-16T11:07:20.320 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T11:07:20.323 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T11:07:20.323 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-16T11:07:20.379 INFO:tasks.workunit:Running workunits matching suites/fsync-tester.sh on client.0... 2024-03-16T11:07:20.380 INFO:tasks.workunit:Running workunit suites/fsync-tester.sh... 2024-03-16T11:07:20.380 DEBUG:teuthology.orchestra.run.smithi078:workunit test suites/fsync-tester.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-16T11:07:20.443 INFO:tasks.workunit.client.0.smithi078.stderr:+ wget http://download.ceph.com/qa/fsync-tester.c 2024-03-16T11:07:20.474 INFO:tasks.workunit.client.0.smithi078.stderr:--2024-03-16 11:07:20-- http://download.ceph.com/qa/fsync-tester.c 2024-03-16T11:07:20.483 INFO:tasks.workunit.client.0.smithi078.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-16T11:07:20.519 INFO:tasks.workunit.client.0.smithi078.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-16T11:07:20.556 INFO:tasks.workunit.client.0.smithi078.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-16T11:07:20.556 INFO:tasks.workunit.client.0.smithi078.stderr:Length: 3024 (3.0K) [application/octet-stream] 2024-03-16T11:07:20.558 INFO:tasks.workunit.client.0.smithi078.stderr:Saving to: ‘fsync-tester.c’ 2024-03-16T11:07:20.561 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:20.561 INFO:tasks.workunit.client.0.smithi078.stderr: 0K .. 100% 621M=0s 2024-03-16T11:07:20.561 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:20.561 INFO:tasks.workunit.client.0.smithi078.stderr:2024-03-16 11:07:20 (621 MB/s) - ‘fsync-tester.c’ saved [3024/3024] 2024-03-16T11:07:20.561 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-16T11:07:20.561 INFO:tasks.workunit.client.0.smithi078.stderr:+ gcc -D_GNU_SOURCE fsync-tester.c -o fsync-tester 2024-03-16T11:07:20.635 INFO:tasks.workunit.client.0.smithi078.stderr:+ ./fsync-tester 2024-03-16T11:07:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:21 smithi098 ceph-mon[35848]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:07:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:21 smithi153 ceph-mon[35521]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:07:21.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:21 smithi078 ceph-mon[32235]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T11:07:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:23 smithi098 ceph-mon[35848]: pgmap v390: 97 pgs: 97 active+clean; 4.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 342 KiB/s wr, 7 op/s 2024-03-16T11:07:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:23 smithi153 ceph-mon[35521]: pgmap v390: 97 pgs: 97 active+clean; 4.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 342 KiB/s wr, 7 op/s 2024-03-16T11:07:23.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:23 smithi078 ceph-mon[32235]: pgmap v390: 97 pgs: 97 active+clean; 4.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 342 KiB/s wr, 7 op/s 2024-03-16T11:07:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:25 smithi098 ceph-mon[35848]: pgmap v391: 97 pgs: 97 active+clean; 33 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.7 MiB/s wr, 6 op/s 2024-03-16T11:07:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:25 smithi153 ceph-mon[35521]: pgmap v391: 97 pgs: 97 active+clean; 33 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.7 MiB/s wr, 6 op/s 2024-03-16T11:07:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:25 smithi078 ceph-mon[32235]: pgmap v391: 97 pgs: 97 active+clean; 33 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.7 MiB/s wr, 6 op/s 2024-03-16T11:07:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:27 smithi098 ceph-mon[35848]: pgmap v392: 97 pgs: 97 active+clean; 85 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 MiB/s wr, 11 op/s 2024-03-16T11:07:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:27 smithi153 ceph-mon[35521]: pgmap v392: 97 pgs: 97 active+clean; 85 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 MiB/s wr, 11 op/s 2024-03-16T11:07:27.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:27 smithi078 ceph-mon[32235]: pgmap v392: 97 pgs: 97 active+clean; 85 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 MiB/s wr, 11 op/s 2024-03-16T11:07:29.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:29 smithi098 ceph-mon[35848]: pgmap v393: 97 pgs: 97 active+clean; 125 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 10 MiB/s wr, 10 op/s 2024-03-16T11:07:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:29 smithi153 ceph-mon[35521]: pgmap v393: 97 pgs: 97 active+clean; 125 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 10 MiB/s wr, 10 op/s 2024-03-16T11:07:29.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:29 smithi078 ceph-mon[32235]: pgmap v393: 97 pgs: 97 active+clean; 125 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 10 MiB/s wr, 10 op/s 2024-03-16T11:07:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:31 smithi098 ceph-mon[35848]: pgmap v394: 97 pgs: 97 active+clean; 190 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 16 MiB/s wr, 12 op/s 2024-03-16T11:07:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:31 smithi153 ceph-mon[35521]: pgmap v394: 97 pgs: 97 active+clean; 190 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 16 MiB/s wr, 12 op/s 2024-03-16T11:07:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:31 smithi078 ceph-mon[32235]: pgmap v394: 97 pgs: 97 active+clean; 190 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 16 MiB/s wr, 12 op/s 2024-03-16T11:07:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 -- 172.21.15.78:0/2335201062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac08101930 msgr2=0x7fac08101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/2335201062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac08101930 0x7fac08101db0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009900 tx=0x7fabfc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 -- 172.21.15.78:0/2335201062 shutdown_connections 2024-03-16T11:07:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/2335201062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac081022f0 0x7fac08108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/2335201062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac08101930 0x7fac08101db0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/2335201062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 0x7fac08100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 -- 172.21.15.78:0/2335201062 >> 172.21.15.78:0/2335201062 conn(0x7fac080fbcb0 msgr2=0x7fac080fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 -- 172.21.15.78:0/2335201062 shutdown_connections 2024-03-16T11:07:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.091+0000 7fac0d4f8640 1 -- 172.21.15.78:0/2335201062 wait complete. 2024-03-16T11:07:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 Processor -- start 2024-03-16T11:07:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 -- start start 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 0x7fac0806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac08101930 0x7fac0819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac081022f0 0x7fac0819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fac0810c2c0 con 0x7fac08101930 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac0810c140 con 0x7fac08100740 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac0d4f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac0810c440 con 0x7fac081022f0 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 0x7fac0806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 0x7fac0806a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:42064/0 (socket says 172.21.15.78:42064) 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac06ffd640 1 -- 172.21.15.78:0/1188538815 learned_addr learned my addr 172.21.15.78:0/1188538815 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac077fe640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac081022f0 0x7fac0819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.092+0000 7fac067fc640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac08101930 0x7fac0819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fac06ffd640 1 -- 172.21.15.78:0/1188538815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac081022f0 msgr2=0x7fac0819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fac06ffd640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac081022f0 0x7fac0819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fac06ffd640 1 -- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac08101930 msgr2=0x7fac0819c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fac06ffd640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac08101930 0x7fac0819c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fac06ffd640 1 -- 172.21.15.78:0/1188538815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0819f5e0 con 0x7fac08100740 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fac06ffd640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 0x7fac0806a4f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fabf800e9c0 tx=0x7fabf800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fabeffff640 1 -- 172.21.15.78:0/1188538815 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf800cd80 con 0x7fac08100740 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fabeffff640 1 -- 172.21.15.78:0/1188538815 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabf80035f0 con 0x7fac08100740 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.093+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac08196580 con 0x7fac08100740 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.094+0000 7fabeffff640 1 -- 172.21.15.78:0/1188538815 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8010690 con 0x7fac08100740 2024-03-16T11:07:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.094+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac08196a40 con 0x7fac08100740 2024-03-16T11:07:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.097+0000 7fabeffff640 1 -- 172.21.15.78:0/1188538815 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fabf8004030 con 0x7fac08100740 2024-03-16T11:07:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.097+0000 7fabeffff640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fabf00a2920 0x7fabf00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.097+0000 7fabeffff640 1 -- 172.21.15.78:0/1188538815 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fabf80cef10 con 0x7fac08100740 2024-03-16T11:07:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.097+0000 7fac067fc640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fabf00a2920 0x7fabf00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 Processor -- start 2024-03-16T11:07:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 -- start start 2024-03-16T11:07:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb10760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb15830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb18330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a0fba75a40 con 0x55a0fbb15ea0 2024-03-16T11:07:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a0fba1e920 con 0x55a0fbb10d30 2024-03-16T11:07:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac101c2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a0fba71ae0 con 0x55a0fbb10340 2024-03-16T11:07:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac067fc640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fabf00a2920 0x7fabf00a4de0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fabfc000c00 tx=0x7fabfc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb10760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb15830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb15830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:42074/0 (socket says 172.21.15.78:42074) 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac067fc640 1 -- 172.21.15.78:0/678670714 learned_addr learned my addr 172.21.15.78:0/678670714 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac077fe640 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb18330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.098+0000 7fac077fe640 1 -- 172.21.15.78:0/678670714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 msgr2=0x55a0fbb10760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fac077fe640 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb10760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fac077fe640 1 -- 172.21.15.78:0/678670714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 msgr2=0x55a0fbb15830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fac077fe640 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb15830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fac077fe640 1 -- 172.21.15.78:0/678670714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0fba36460 con 0x55a0fbb15ea0 2024-03-16T11:07:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fac077fe640 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb18330 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fac00009520 tx=0x7fac000099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600a55460ca122fe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fabed7fa640 1 -- 172.21.15.78:0/678670714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00004b20 con 0x55a0fbb15ea0 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fabed7fa640 1 -- 172.21.15.78:0/678670714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac00004cc0 con 0x55a0fbb15ea0 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.099+0000 7fabed7fa640 1 -- 172.21.15.78:0/678670714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00005020 con 0x55a0fbb15ea0 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.100+0000 7fac101c2740 1 -- 172.21.15.78:0/678670714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 msgr2=0x55a0fbb18330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.100+0000 7fac101c2740 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb18330 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fac00009520 tx=0x7fac000099e0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 -- 172.21.15.78:0/678670714 shutdown_connections 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb18330 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb15830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 --2- 172.21.15.78:0/678670714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb10760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 -- 172.21.15.78:0/678670714 >> 172.21.15.78:0/678670714 conn(0x55a0fbb0b8b0 msgr2=0x55a0fbb0bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 -- 172.21.15.78:0/678670714 shutdown_connections 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 -- 172.21.15.78:0/678670714 wait complete. 2024-03-16T11:07:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 -- start start 2024-03-16T11:07:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.101+0000 7fac101c2740 1 Processor -- start 2024-03-16T11:07:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac101c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb1a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac101c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac101c2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac101c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a0fbb18e50 con 0x55a0fbb10d30 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac101c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a0fba31d70 con 0x55a0fbb10340 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac101c2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a0fbb18fd0 con 0x55a0fbb15ea0 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac101c2740 10 client.0 _ll_register_callbacks cb 0x55a0fb789a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:59750/0 (socket says 172.21.15.78:59750) 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 -- 172.21.15.78:0/3781070008 learned_addr learned my addr 172.21.15.78:0/3781070008 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac06ffd640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb1a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac067fc640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 -- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 msgr2=0x55a0fbb1a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb1a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 -- 172.21.15.78:0/3781070008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 msgr2=0x55a0fbb1ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.102+0000 7fac077fe640 1 -- 172.21.15.78:0/3781070008 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0fb78a210 con 0x55a0fbb15ea0 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fac06ffd640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb1a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fac067fc640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fac077fe640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fac00004270 tx=0x7fac00004350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d76cc4e8ee3793d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fabeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fabeaff5640 1 -- 172.21.15.78:0/3781070008 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00027070 con 0x55a0fbb15ea0 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac00004cc0 con 0x55a0fbb15ea0 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fabeaff5640 1 -- 172.21.15.78:0/3781070008 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fac00003c40 con 0x55a0fbb15ea0 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fabeaff5640 1 -- 172.21.15.78:0/3781070008 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fac00033050 con 0x55a0fbb15ea0 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fac101c2740 10 client.25621 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.103+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a0fb856620 con 0x55a0fbb15ea0 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fabeaff5640 1 -- 172.21.15.78:0/3781070008 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac00003e40 con 0x55a0fbb15ea0 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fabeaff5640 1 -- 172.21.15.78:0/3781070008 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fac00023420 con 0x55a0fbb15ea0 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fac101c2740 10 client.25621 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fac101c2740 10 client.25621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fac101c2740 20 client.25621 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fac101c2740 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb26180 0x55a0fbb28570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fac101c2740 4 client.25621 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fac00004cc0 con 0x55a0fbb26180 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.104+0000 7fac06ffd640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb26180 0x55a0fbb28570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.105+0000 7fac06ffd640 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb26180 0x55a0fbb28570 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53376644e75a3ae7 server_cookie=407d7c53dae5f362 in_seq=0 out_seq=0 2024-03-16T11:07:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.105+0000 7fabeaff5640 10 client.25621 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.106+0000 7fabeaff5640 1 -- 172.21.15.78:0/3781070008 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fac00004cc0 con 0x55a0fbb26180 2024-03-16T11:07:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.106+0000 7fabeaff5640 10 client.25621 handle_command_reply: tid=0 2024-03-16T11:07:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.106+0000 7fac101c2740 1 client.25621 shutdown 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 msgr2=0x55a0fbb21ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fac00004270 tx=0x7fac00004350 comp rx=0 tx=0).stop 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 shutdown_connections 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb26180 0x55a0fbb28570 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21ba0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 --2- 172.21.15.78:0/3781070008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb1a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 >> 172.21.15.78:0/3781070008 conn(0x55a0fbb0b8b0 msgr2=0x55a0fbb0abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fabeaff5640 0 client.25621 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 shutdown_connections 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.107+0000 7fac101c2740 1 -- 172.21.15.78:0/3781070008 wait complete. 2024-03-16T11:07:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.108+0000 7fac101c2740 20 client.25621 trim_cache size 0 max 16384 2024-03-16T11:07:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.130+0000 7fac101c2740 1 Processor -- start 2024-03-16T11:07:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac101c2740 1 -- start start 2024-03-16T11:07:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac101c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb11ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac101c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb12530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb12530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb11ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb12530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39522/0 (socket says 172.21.15.78:39522) 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb11ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:42090/0 (socket says 172.21.15.78:42090) 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac067fc640 1 -- 172.21.15.78:0/918189181 learned_addr learned my addr 172.21.15.78:0/918189181 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac101c2740 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fac081a92e0 con 0x55a0fbb10d30 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac081a6bb0 con 0x55a0fbb10340 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.131+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac081a49e0 con 0x55a0fbb15ea0 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fac077fe640 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fac067fc640 1 -- 172.21.15.78:0/918189181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 msgr2=0x55a0fbb21b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fac067fc640 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fac067fc640 1 -- 172.21.15.78:0/918189181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 msgr2=0x55a0fbb11ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fac067fc640 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb11ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fac067fc640 1 -- 172.21.15.78:0/918189181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0fbb07310 con 0x55a0fbb10d30 2024-03-16T11:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fac067fc640 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb12530 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fabfc02c240 tx=0x7fabfc03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35677149dee05778 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fabecff9640 1 -- 172.21.15.78:0/918189181 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc051030 con 0x55a0fbb10d30 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.132+0000 7fabecff9640 1 -- 172.21.15.78:0/918189181 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabfc0046c0 con 0x55a0fbb10d30 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.133+0000 7fabecff9640 1 -- 172.21.15.78:0/918189181 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0035f0 con 0x55a0fbb10d30 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.133+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 msgr2=0x55a0fbb12530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.133+0000 7fac101c2740 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb12530 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fabfc02c240 tx=0x7fabfc03e000 comp rx=0 tx=0).stop 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 shutdown_connections 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb21b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb10d30 0x55a0fbb12530 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 --2- 172.21.15.78:0/918189181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10340 0x55a0fbb11ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 >> 172.21.15.78:0/918189181 conn(0x55a0fbb0b8b0 msgr2=0x55a0fbb0bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 shutdown_connections 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 -- 172.21.15.78:0/918189181 wait complete. 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.134+0000 7fac101c2740 1 -- start start 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 1 Processor -- start 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb1f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb20bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fac081a5070 con 0x55a0fbb15ea0 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac081a64e0 con 0x55a0fbb10d30 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac081a5740 con 0x55a0fbb10340 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac101c2740 10 client.0 _ll_register_callbacks cb 0x7fabfc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb1f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac067fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:42102/0 (socket says 172.21.15.78:42102) 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac067fc640 1 -- 172.21.15.78:0/788178405 learned_addr learned my addr 172.21.15.78:0/788178405 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.135+0000 7fac077fe640 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb20bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac06ffd640 1 -- 172.21.15.78:0/788178405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 msgr2=0x55a0fbb1fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac06ffd640 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac06ffd640 1 -- 172.21.15.78:0/788178405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 msgr2=0x55a0fbb20bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac06ffd640 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb20bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac06ffd640 1 -- 172.21.15.78:0/788178405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0fbb211e0 con 0x55a0fbb10340 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac077fe640 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb20bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac06ffd640 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb1f740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fabf80cca10 tx=0x7fabf8086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=939119c16821d958 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fabeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fabeaff5640 1 -- 172.21.15.78:0/788178405 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf809fa00 con 0x55a0fbb10340 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fabeaff5640 1 -- 172.21.15.78:0/788178405 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fabf8004320 con 0x55a0fbb10340 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.136+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabfc0046c0 con 0x55a0fbb10340 2024-03-16T11:07:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fabeaff5640 1 -- 172.21.15.78:0/788178405 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fabf80a4050 con 0x55a0fbb10340 2024-03-16T11:07:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 10 client.25627 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a0fbb8c990 con 0x55a0fbb10340 2024-03-16T11:07:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fabeaff5640 1 -- 172.21.15.78:0/788178405 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabfc0046c0 con 0x55a0fbb10340 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fabeaff5640 1 -- 172.21.15.78:0/788178405 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fabf8097040 con 0x55a0fbb10340 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 10 client.25627 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 10 client.25627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 20 client.25627 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb8d340 0x55a0fbb8f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 4 client.25627 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0fba82f80 con 0x55a0fbb8d340 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.137+0000 7fac067fc640 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb8d340 0x55a0fbb8f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.138+0000 7fabeaff5640 10 client.25627 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.138+0000 7fac067fc640 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb8d340 0x55a0fbb8f760 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36bddd7ffb3b4273 server_cookie=985378723ec5d064 in_seq=0 out_seq=0 2024-03-16T11:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.151+0000 7fabeaff5640 1 -- 172.21.15.78:0/788178405 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0fba82f80 con 0x55a0fbb8d340 2024-03-16T11:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.151+0000 7fabeaff5640 10 client.25627 handle_command_reply: tid=0 2024-03-16T11:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.151+0000 7fac101c2740 1 client.25627 shutdown 2024-03-16T11:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.151+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 msgr2=0x55a0fbb1f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.151+0000 7fac101c2740 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb1f740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fabf80cca10 tx=0x7fabf8086040 comp rx=0 tx=0).stop 2024-03-16T11:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.151+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 shutdown_connections 2024-03-16T11:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.151+0000 7fac101c2740 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55a0fbb8d340 0x55a0fbb8f760 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fabeaff5640 0 client.25627 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fac101c2740 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a0fbb15ea0 0x55a0fbb20bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fac101c2740 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0fbb10d30 0x55a0fbb1fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fac101c2740 1 --2- 172.21.15.78:0/788178405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a0fbb10340 0x55a0fbb1f740 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 >> 172.21.15.78:0/788178405 conn(0x55a0fbb0b8b0 msgr2=0x55a0fbb11ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 shutdown_connections 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fac101c2740 1 -- 172.21.15.78:0/788178405 wait complete. 2024-03-16T11:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.152+0000 7fac101c2740 20 client.25627 trim_cache size 0 max 16384 2024-03-16T11:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fabf00a2920 msgr2=0x7fabf00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fabf00a2920 0x7fabf00a4de0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fabfc000c00 tx=0x7fabfc02c7e0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 msgr2=0x7fac0806a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 0x7fac0806a4f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fabf800e9c0 tx=0x7fabf800ee80 comp rx=0 tx=0).stop 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 shutdown_connections 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fabf00a2920 0x7fabf00a4de0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac081022f0 0x7fac0819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac08101930 0x7fac0819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 --2- 172.21.15.78:0/1188538815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac08100740 0x7fac0806a4f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 >> 172.21.15.78:0/1188538815 conn(0x7fac080fbcb0 msgr2=0x7fac08104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 shutdown_connections 2024-03-16T11:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:32.153+0000 7fac0d4f8640 1 -- 172.21.15.78:0/1188538815 wait complete. 2024-03-16T11:07:33.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:33 smithi098 ceph-mon[35848]: pgmap v395: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 21 MiB/s wr, 251 op/s 2024-03-16T11:07:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:33 smithi153 ceph-mon[35521]: pgmap v395: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 21 MiB/s wr, 251 op/s 2024-03-16T11:07:33.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:33 smithi078 ceph-mon[32235]: pgmap v395: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 21 MiB/s wr, 251 op/s 2024-03-16T11:07:35.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:07:35.300 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:07:35.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.453+0000 7f08e1a34640 1 -- 172.21.15.78:0/2414009535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 msgr2=0x7f08dc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.453+0000 7f08e1a34640 1 --2- 172.21.15.78:0/2414009535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc101d50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009900 tx=0x7f08cc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.453+0000 7f08e1a34640 1 -- 172.21.15.78:0/2414009535 shutdown_connections 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.453+0000 7f08e1a34640 1 --2- 172.21.15.78:0/2414009535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08dc102290 0x7f08dc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.453+0000 7f08e1a34640 1 --2- 172.21.15.78:0/2414009535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc101d50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.453+0000 7f08e1a34640 1 --2- 172.21.15.78:0/2414009535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc1006e0 0x7f08dc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.453+0000 7f08e1a34640 1 -- 172.21.15.78:0/2414009535 >> 172.21.15.78:0/2414009535 conn(0x7f08dc0fbc70 msgr2=0x7f08dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.454+0000 7f08e1a34640 1 -- 172.21.15.78:0/2414009535 shutdown_connections 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.454+0000 7f08e1a34640 1 -- 172.21.15.78:0/2414009535 wait complete. 2024-03-16T11:07:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.454+0000 7f08e1a34640 1 Processor -- start 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.454+0000 7f08e1a34640 1 -- start start 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08e1a34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc1006e0 0x7f08dc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08e1a34640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08e1a34640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08dc102290 0x7f08dc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08e1a34640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc10d370 con 0x7f08dc1018d0 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08e1a34640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc10d1f0 con 0x7f08dc102290 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08e1a34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc10d4f0 con 0x7f08dc1006e0 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50204/0 (socket says 172.21.15.78:50204) 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 -- 172.21.15.78:0/4122103715 learned_addr learned my addr 172.21.15.78:0/4122103715 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08db7fe640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08dc102290 0x7f08dc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08daffd640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc1006e0 0x7f08dc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 -- 172.21.15.78:0/4122103715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc1006e0 msgr2=0x7f08dc197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc1006e0 0x7f08dc197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 -- 172.21.15.78:0/4122103715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08dc102290 msgr2=0x7f08dc19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08dc102290 0x7f08dc19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.455+0000 7f08da7fc640 1 -- 172.21.15.78:0/4122103715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc19f650 con 0x7f08dc1018d0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.456+0000 7f08da7fc640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc197f70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f08cc000c00 tx=0x7f08cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.456+0000 7f08e0a32640 1 -- 172.21.15.78:0/4122103715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc043020 con 0x7f08dc1018d0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.456+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08dc19f8e0 con 0x7f08dc1018d0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.456+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08dc19fe80 con 0x7f08dc1018d0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.456+0000 7f08e0a32640 1 -- 172.21.15.78:0/4122103715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08cc0043c0 con 0x7f08dc1018d0 2024-03-16T11:07:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.456+0000 7f08e0a32640 1 -- 172.21.15.78:0/4122103715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc003bf0 con 0x7f08dc1018d0 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 Processor -- start 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e0a32640 1 -- 172.21.15.78:0/4122103715 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f08cc046020 con 0x7f08dc1018d0 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 -- start start 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155eba510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ebd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ecaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x565155e1f500 con 0x565155eba0f0 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x565155dc7be0 con 0x565155ebaae0 2024-03-16T11:07:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.460+0000 7f08e461a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565155e1b5a0 con 0x565155ebdc40 2024-03-16T11:07:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ebd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ebd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:40850/0 (socket says 172.21.15.78:40850) 2024-03-16T11:07:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 -- 172.21.15.78:0/3996882359 learned_addr learned my addr 172.21.15.78:0/3996882359 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08db7fe640 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155eba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08e0a32640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f08c40a28d0 0x7f08c40a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 -- 172.21.15.78:0/3996882359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 msgr2=0x565155ecaab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ecaab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 -- 172.21.15.78:0/3996882359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 msgr2=0x565155eba510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155eba510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08daffd640 1 -- 172.21.15.78:0/3996882359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565155ddb4b0 con 0x565155ebaae0 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08db7fe640 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155eba510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.461+0000 7f08e0a32640 1 -- 172.21.15.78:0/4122103715 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f08cc0f0b30 con 0x7f08dc1018d0 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.462+0000 7f08daffd640 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ebd5d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009ca0 tx=0x7f08d0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=684b8e184f8f0e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.462+0000 7f08db7fe640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f08c40a28d0 0x7f08c40a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.462+0000 7f08c1ffb640 1 -- 172.21.15.78:0/3996882359 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0004220 con 0x565155ebaae0 2024-03-16T11:07:35.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.462+0000 7f08c1ffb640 1 -- 172.21.15.78:0/3996882359 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08d00043c0 con 0x565155ebaae0 2024-03-16T11:07:35.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.462+0000 7f08c1ffb640 1 -- 172.21.15.78:0/3996882359 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0019890 con 0x565155ebaae0 2024-03-16T11:07:35.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:35 smithi098 ceph-mon[35848]: pgmap v396: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 25 MiB/s wr, 773 op/s 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 -- 172.21.15.78:0/3996882359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 msgr2=0x565155ebd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ebd5d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009ca0 tx=0x7f08d0011810 comp rx=0 tx=0).stop 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08db7fe640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f08c40a28d0 0x7f08c40a4d90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x565155ebb340 tx=0x7f08d4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 -- 172.21.15.78:0/3996882359 shutdown_connections 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ecaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ebd5d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 --2- 172.21.15.78:0/3996882359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155eba510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 -- 172.21.15.78:0/3996882359 >> 172.21.15.78:0/3996882359 conn(0x565155eb56e0 msgr2=0x565155eb5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 -- 172.21.15.78:0/3996882359 shutdown_connections 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 -- 172.21.15.78:0/3996882359 wait complete. 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.463+0000 7f08e461a740 1 -- start start 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 1 Processor -- start 2024-03-16T11:07:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155eccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebaae0 0x565155ecd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ed44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x565155ecb500 con 0x565155ebaae0 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x565155ddb160 con 0x565155eba0f0 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565155ecb680 con 0x565155ebdc40 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08e461a740 10 client.0 _ll_register_callbacks cb 0x565155dbf150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08db7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ed44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08db7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ed44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57200/0 (socket says 172.21.15.78:57200) 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.464+0000 7f08db7fe640 1 -- 172.21.15.78:0/3490558955 learned_addr learned my addr 172.21.15.78:0/3490558955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08daffd640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155eccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08da7fc640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebaae0 0x565155ecd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08db7fe640 1 -- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 msgr2=0x565155eccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08db7fe640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155eccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08db7fe640 1 -- 172.21.15.78:0/3490558955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebaae0 msgr2=0x565155ecd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08db7fe640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebaae0 0x565155ecd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08db7fe640 1 -- 172.21.15.78:0/3490558955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565155bfd7d0 con 0x565155ebdc40 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08da7fc640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebaae0 0x565155ecd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08daffd640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155eccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08db7fe640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ed44e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08d400ffd0 tx=0x7f08d4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46fc89763f586d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08baffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08baffd640 1 -- 172.21.15.78:0/3490558955 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4004d20 con 0x565155ebdc40 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08baffd640 1 -- 172.21.15.78:0/3490558955 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f08d4003970 con 0x565155ebdc40 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.465+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08d00043c0 con 0x565155ebdc40 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.466+0000 7f08baffd640 1 -- 172.21.15.78:0/3490558955 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f08d4003b70 con 0x565155ebdc40 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.466+0000 7f08e461a740 10 client.25636 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.466+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x565155b333f0 con 0x565155ebdc40 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.466+0000 7f08baffd640 1 -- 172.21.15.78:0/3490558955 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08d00043c0 con 0x565155ebdc40 2024-03-16T11:07:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08baffd640 1 -- 172.21.15.78:0/3490558955 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f08d401f630 con 0x565155ebdc40 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08e461a740 10 client.25636 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08e461a740 10 client.25636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08e461a740 20 client.25636 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08e461a740 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155ed8b20 0x565155edaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08e461a740 4 client.25636 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565155e2ca40 con 0x565155ed8b20 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.467+0000 7f08daffd640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155ed8b20 0x565155edaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.468+0000 7f08baffd640 10 client.25636 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.468+0000 7f08daffd640 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155ed8b20 0x565155edaf40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f4515e2a42d4769 server_cookie=7d3d083deedde4a1 in_seq=0 out_seq=0 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.468+0000 7f08baffd640 1 -- 172.21.15.78:0/3490558955 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565155e2ca40 con 0x565155ed8b20 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.468+0000 7f08baffd640 10 client.25636 handle_command_reply: tid=0 2024-03-16T11:07:35.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 client.25636 shutdown 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 msgr2=0x565155ed44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ed44e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08d400ffd0 tx=0x7f08d4004a40 comp rx=0 tx=0).stop 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 shutdown_connections 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155ed8b20 0x565155edaf40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155ed44e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebaae0 0x565155ecd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 --2- 172.21.15.78:0/3490558955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155eccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 >> 172.21.15.78:0/3490558955 conn(0x565155eb56e0 msgr2=0x565155eb4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08baffd640 0 client.25636 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 shutdown_connections 2024-03-16T11:07:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 1 -- 172.21.15.78:0/3490558955 wait complete. 2024-03-16T11:07:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.469+0000 7f08e461a740 20 client.25636 trim_cache size 0 max 16384 2024-03-16T11:07:35.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.494+0000 7f08e461a740 1 Processor -- start 2024-03-16T11:07:35.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08e461a740 1 -- start start 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08e461a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155ecc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08e461a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ed4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08e461a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155eb4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08e461a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8010d90 con 0x565155eba0f0 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155ecc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08e461a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08c800e660 con 0x565155ebaae0 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155ecc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50226/0 (socket says 172.21.15.78:50226) 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 -- 172.21.15.78:0/644033686 learned_addr learned my addr 172.21.15.78:0/644033686 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08da7fc640 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ed4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08e461a740 1 -- 172.21.15.78:0/644033686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08c800c490 con 0x565155ebdc40 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08db7fe640 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155eb4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 -- 172.21.15.78:0/644033686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 msgr2=0x565155eb4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155eb4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 -- 172.21.15.78:0/644033686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 msgr2=0x565155ed4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ed4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08daffd640 1 -- 172.21.15.78:0/644033686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565155af30f0 con 0x565155eba0f0 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.495+0000 7f08da7fc640 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ed4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.496+0000 7f08daffd640 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155ecc940 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009ca0 tx=0x7f08d0003d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdd77d4225a356a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.496+0000 7f08c17fa640 1 -- 172.21.15.78:0/644033686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d00097a0 con 0x565155eba0f0 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.496+0000 7f08c17fa640 1 -- 172.21.15.78:0/644033686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08d00035f0 con 0x565155eba0f0 2024-03-16T11:07:35.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.496+0000 7f08c17fa640 1 -- 172.21.15.78:0/644033686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d002c250 con 0x565155eba0f0 2024-03-16T11:07:35.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 -- 172.21.15.78:0/644033686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 msgr2=0x565155ecc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155ecc940 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009ca0 tx=0x7f08d0003d20 comp rx=0 tx=0).stop 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 -- 172.21.15.78:0/644033686 shutdown_connections 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebdc40 0x565155eb4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155ebaae0 0x565155ed4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 --2- 172.21.15.78:0/644033686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155eba0f0 0x565155ecc940 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 -- 172.21.15.78:0/644033686 >> 172.21.15.78:0/644033686 conn(0x565155eb56e0 msgr2=0x565155ecc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 -- 172.21.15.78:0/644033686 shutdown_connections 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.497+0000 7f08e461a740 1 -- 172.21.15.78:0/644033686 wait complete. 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 -- start start 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 Processor -- start 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155ed1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebaae0 0x565155ed23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebdc40 0x565155ed3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08c800cb20 con 0x565155ebdc40 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08c800df90 con 0x565155eba0f0 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08c800d1f0 con 0x565155ebaae0 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08e461a740 10 client.0 _ll_register_callbacks cb 0x7f08d4011ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08da7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebaae0 0x565155ed23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08da7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebaae0 0x565155ed23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57228/0 (socket says 172.21.15.78:57228) 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08da7fc640 1 -- 172.21.15.78:0/2798760068 learned_addr learned my addr 172.21.15.78:0/2798760068 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08daffd640 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155ed1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.498+0000 7f08db7fe640 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebdc40 0x565155ed3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08da7fc640 1 -- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 msgr2=0x565155ed1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08da7fc640 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155ed1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08da7fc640 1 -- 172.21.15.78:0/2798760068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebdc40 msgr2=0x565155ed3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08da7fc640 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebdc40 0x565155ed3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08da7fc640 1 -- 172.21.15.78:0/2798760068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565155de69d0 con 0x565155ebaae0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08da7fc640 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebaae0 0x565155ed23c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0bd1d0 tx=0x7f08cc0bd2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc222df51f695bc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08baffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08baffd640 1 -- 172.21.15.78:0/2798760068 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc0bb610 con 0x565155ebaae0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.499+0000 7f08baffd640 1 -- 172.21.15.78:0/2798760068 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f08cc0c59e0 con 0x565155ebaae0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08d00035f0 con 0x565155ebaae0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08baffd640 1 -- 172.21.15.78:0/2798760068 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f08cc0bdbd0 con 0x565155ebaae0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 10 client.25642 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x565155f3eed0 con 0x565155ebaae0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08baffd640 1 -- 172.21.15.78:0/2798760068 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08cc0bd4d0 con 0x565155ebaae0 2024-03-16T11:07:35.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08baffd640 1 -- 172.21.15.78:0/2798760068 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f08cc0c11f0 con 0x565155ebaae0 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 10 client.25642 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 20 client.25642 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155f3f8b0 0x565155f41cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 4 client.25642 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.500+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f08d00035f0 con 0x565155f3f8b0 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.501+0000 7f08daffd640 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155f3f8b0 0x565155f41cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.501+0000 7f08baffd640 10 client.25642 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.501+0000 7f08daffd640 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155f3f8b0 0x565155f41cd0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bf10eb14e352d40 server_cookie=72f5dd8a0b71660e in_seq=0 out_seq=0 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.501+0000 7f08baffd640 1 -- 172.21.15.78:0/2798760068 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f08d00035f0 con 0x565155f3f8b0 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.501+0000 7f08baffd640 10 client.25642 handle_command_reply: tid=0 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 client.25642 shutdown 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebaae0 msgr2=0x565155ed23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebaae0 0x565155ed23c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0bd1d0 tx=0x7f08cc0bd2a0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 shutdown_connections 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x565155f3f8b0 0x565155f41cd0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565155ebdc40 0x565155ed3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565155ebaae0 0x565155ed23c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 --2- 172.21.15.78:0/2798760068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565155eba0f0 0x565155ed1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 >> 172.21.15.78:0/2798760068 conn(0x565155eb56e0 msgr2=0x565155eb8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08baffd640 0 client.25642 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 shutdown_connections 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.502+0000 7f08e461a740 1 -- 172.21.15.78:0/2798760068 wait complete. 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.503+0000 7f08e461a740 20 client.25642 trim_cache size 0 max 16384 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:07:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-16T11:07:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-16T11:07:35.507 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:07:35.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.503+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f08c40a28d0 msgr2=0x7f08c40a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.503+0000 7f08e1a34640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f08c40a28d0 0x7f08c40a4d90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x565155ebb340 tx=0x7f08d4004ab0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 msgr2=0x7f08dc197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc197f70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f08cc000c00 tx=0x7f08cc004040 comp rx=0 tx=0).stop 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 shutdown_connections 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08dc102290 0x7f08dc19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08dc1018d0 0x7f08dc197f70 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f08c40a28d0 0x7f08c40a4d90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 --2- 172.21.15.78:0/4122103715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc1006e0 0x7f08dc197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 >> 172.21.15.78:0/4122103715 conn(0x7f08dc0fbc70 msgr2=0x7f08dc104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 shutdown_connections 2024-03-16T11:07:35.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.504+0000 7f08e1a34640 1 -- 172.21.15.78:0/4122103715 wait complete. 2024-03-16T11:07:35.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:35 smithi078 ceph-mon[32235]: pgmap v396: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 25 MiB/s wr, 773 op/s 2024-03-16T11:07:35.521 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T11:07:35.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:07:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:35 smithi153 ceph-mon[35521]: pgmap v396: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 25 MiB/s wr, 773 op/s 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 -- 172.21.15.78:0/1499420498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 msgr2=0x7f0fa0104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 --2- 172.21.15.78:0/1499420498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 0x7f0fa0104e60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009be0 tx=0x7f0f9802bef0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 -- 172.21.15.78:0/1499420498 shutdown_connections 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 --2- 172.21.15.78:0/1499420498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa01053a0 0x7f0fa0105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 --2- 172.21.15.78:0/1499420498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 0x7f0fa0104e60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 --2- 172.21.15.78:0/1499420498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa0100080 0x7f0fa01024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 -- 172.21.15.78:0/1499420498 >> 172.21.15.78:0/1499420498 conn(0x7f0fa00fbcc0 msgr2=0x7f0fa00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 -- 172.21.15.78:0/1499420498 shutdown_connections 2024-03-16T11:07:35.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.672+0000 7f0fa5990640 1 -- 172.21.15.78:0/1499420498 wait complete. 2024-03-16T11:07:35.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.673+0000 7f0fa5990640 1 Processor -- start 2024-03-16T11:07:35.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.673+0000 7f0fa5990640 1 -- start start 2024-03-16T11:07:35.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0fa5990640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa0100080 0x7f0fa019e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0fa5990640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 0x7f0fa019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0fa5990640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa01053a0 0x7f0fa01a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0fa5990640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa0113910 con 0x7f0fa0100080 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0fa5990640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa0113790 con 0x7f0fa01029f0 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0fa5990640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa0113a90 con 0x7f0fa01053a0 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa0100080 0x7f0fa019e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 0x7f0fa019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa0100080 0x7f0fa019e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50234/0 (socket says 172.21.15.78:50234) 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 -- 172.21.15.78:0/4251797894 learned_addr learned my addr 172.21.15.78:0/4251797894 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9f7fe640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa01053a0 0x7f0fa01a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 -- 172.21.15.78:0/4251797894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa01053a0 msgr2=0x7f0fa01a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa01053a0 0x7f0fa01a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 -- 172.21.15.78:0/4251797894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 msgr2=0x7f0fa019e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 0x7f0fa019e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.674+0000 7f0f9effd640 1 -- 172.21.15.78:0/4251797894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa01a5c60 con 0x7f0fa0100080 2024-03-16T11:07:35.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.675+0000 7f0f9effd640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa0100080 0x7f0fa019e040 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000e9f0 tx=0x7f0f9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.675+0000 7f0fa498e640 1 -- 172.21.15.78:0/4251797894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9000cd80 con 0x7f0fa0100080 2024-03-16T11:07:35.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.675+0000 7f0fa498e640 1 -- 172.21.15.78:0/4251797894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f900108e0 con 0x7f0fa0100080 2024-03-16T11:07:35.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.675+0000 7f0fa498e640 1 -- 172.21.15.78:0/4251797894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f90018920 con 0x7f0fa0100080 2024-03-16T11:07:35.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.675+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa01a5f50 con 0x7f0fa0100080 2024-03-16T11:07:35.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.675+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa01a6360 con 0x7f0fa0100080 2024-03-16T11:07:35.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.677+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fa00fd7b0 con 0x7f0fa0100080 2024-03-16T11:07:35.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.680+0000 7f0fa498e640 1 -- 172.21.15.78:0/4251797894 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0f90010a90 con 0x7f0fa0100080 2024-03-16T11:07:35.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.681+0000 7f0fa498e640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0f880a2880 0x7f0f880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:35.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.681+0000 7f0fa498e640 1 -- 172.21.15.78:0/4251797894 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0f900cf1c0 con 0x7f0fa0100080 2024-03-16T11:07:35.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.681+0000 7f0f9e7fc640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0f880a2880 0x7f0f880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:35.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.682+0000 7f0f9e7fc640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0f880a2880 0x7f0f880a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f98004af0 tx=0x7f0f98034f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:35.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:35.685+0000 7f0fa498e640 1 -- 172.21.15.78:0/4251797894 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f9009bbd0 con 0x7f0fa0100080 2024-03-16T11:07:36.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.047+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fa002d460 con 0x7f0fa0100080 2024-03-16T11:07:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.048+0000 7f0fa498e640 1 -- 172.21.15.78:0/4251797894 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7f0f9009fc70 con 0x7f0fa0100080 2024-03-16T11:07:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:07:36.048 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:07:36.049 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:07:36.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0f880a2880 msgr2=0x7f0f880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0f880a2880 0x7f0f880a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f98004af0 tx=0x7f0f98034f90 comp rx=0 tx=0).stop 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa0100080 msgr2=0x7f0fa019e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa0100080 0x7f0fa019e040 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000e9f0 tx=0x7f0f9000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 shutdown_connections 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0f880a2880 0x7f0f880a4d40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa01053a0 0x7f0fa01a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fa01029f0 0x7f0fa019e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 --2- 172.21.15.78:0/4251797894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fa0100080 0x7f0fa019e040 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 >> 172.21.15.78:0/4251797894 conn(0x7f0fa00fbcc0 msgr2=0x7f0fa0100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 shutdown_connections 2024-03-16T11:07:36.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.051+0000 7f0fa5990640 1 -- 172.21.15.78:0/4251797894 wait complete. 2024-03-16T11:07:36.067 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T11:07:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 -- 172.21.15.78:0/269374092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08104c30 msgr2=0x7fdb08107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 --2- 172.21.15.78:0/269374092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08104c30 0x7fdb08107020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009900 tx=0x7fdb0402bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 -- 172.21.15.78:0/269374092 shutdown_connections 2024-03-16T11:07:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 --2- 172.21.15.78:0/269374092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08107560 0x7fdb08109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 --2- 172.21.15.78:0/269374092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08104c30 0x7fdb08107020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 --2- 172.21.15.78:0/269374092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb08102300 0x7fdb081046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 -- 172.21.15.78:0/269374092 >> 172.21.15.78:0/269374092 conn(0x7fdb080fbc70 msgr2=0x7fdb080fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 -- 172.21.15.78:0/269374092 shutdown_connections 2024-03-16T11:07:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.212+0000 7fdb0f337640 1 -- 172.21.15.78:0/269374092 wait complete. 2024-03-16T11:07:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 Processor -- start 2024-03-16T11:07:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 -- start start 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08102300 0x7fdb08101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb08104c30 0x7fdb081002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08107560 0x7fdb081007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0810d2e0 con 0x7fdb08104c30 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0810d160 con 0x7fdb08102300 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0f337640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0810d460 con 0x7fdb08107560 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0d0d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08102300 0x7fdb08101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0c8d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb08104c30 0x7fdb081002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0d0d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08102300 0x7fdb08101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:40928/0 (socket says 172.21.15.78:40928) 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/2239873960 learned_addr learned my addr 172.21.15.78:0/2239873960 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.213+0000 7fdb0d8d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08107560 0x7fdb081007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdb0c8d2640 1 -- 172.21.15.78:0/2239873960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08107560 msgr2=0x7fdb081007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08107560 0x7fdb081007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdb0c8d2640 1 -- 172.21.15.78:0/2239873960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08102300 msgr2=0x7fdb08101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08102300 0x7fdb08101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdb0c8d2640 1 -- 172.21.15.78:0/2239873960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb08101080 con 0x7fdb08104c30 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08102300 0x7fdb08101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb08104c30 0x7fdb081002b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdb04036040 tx=0x7fdb04005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.214+0000 7fdafe7fc640 1 -- 172.21.15.78:0/2239873960 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04043020 con 0x7fdb08104c30 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.215+0000 7fdafe7fc640 1 -- 172.21.15.78:0/2239873960 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb04004350 con 0x7fdb08104c30 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.215+0000 7fdafe7fc640 1 -- 172.21.15.78:0/2239873960 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04003ab0 con 0x7fdb08104c30 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.215+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb081a1ba0 con 0x7fdb08104c30 2024-03-16T11:07:36.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.215+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb081a2070 con 0x7fdb08104c30 2024-03-16T11:07:36.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.217+0000 7fdafe7fc640 1 -- 172.21.15.78:0/2239873960 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fdb04046020 con 0x7fdb08104c30 2024-03-16T11:07:36.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.218+0000 7fdb11ffc740 1 Processor -- start 2024-03-16T11:07:36.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.218+0000 7fdb11ffc740 1 -- start start 2024-03-16T11:07:36.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.218+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc2850 0x560bc0dc2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dc7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb11ffc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560bc0d27490 con 0x560bc0dc3240 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb11ffc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bc0ccfc90 con 0x560bc0dc83b0 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb11ffc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560bc0d23530 con 0x560bc0dc2850 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdafe7fc640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdaf40a2900 0x7fdaf40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc2850 0x560bc0dc2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc2850 0x560bc0dc2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57274/0 (socket says 172.21.15.78:57274) 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 -- 172.21.15.78:0/518243428 learned_addr learned my addr 172.21.15.78:0/518243428 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dc7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdaf40a2900 0x7fdaf40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdafe7fc640 1 -- 172.21.15.78:0/2239873960 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdb040f0b00 con 0x7fdb08104c30 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 -- 172.21.15.78:0/518243428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 msgr2=0x560bc0dca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 -- 172.21.15.78:0/518243428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 msgr2=0x560bc0dc7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dc7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d8d4640 1 -- 172.21.15.78:0/518243428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc0d8a500 con 0x560bc0dc2850 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dc7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.219+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.220+0000 7fdb0d8d4640 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc2850 0x560bc0dc2c70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb00012e50 tx=0x7fdb00010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f960fe9ab8722c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.220+0000 7fdb0d8d4640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdaf40a2900 0x7fdaf40a4dc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009370 tx=0x7fdb00017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.220+0000 7fdaf37fe640 1 -- 172.21.15.78:0/518243428 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb00012930 con 0x560bc0dc2850 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.220+0000 7fdaf37fe640 1 -- 172.21.15.78:0/518243428 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb00003e20 con 0x560bc0dc2850 2024-03-16T11:07:36.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.220+0000 7fdaf37fe640 1 -- 172.21.15.78:0/518243428 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb000042c0 con 0x560bc0dc2850 2024-03-16T11:07:36.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 -- 172.21.15.78:0/518243428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc2850 msgr2=0x560bc0dc2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc2850 0x560bc0dc2c70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb00012e50 tx=0x7fdb00010740 comp rx=0 tx=0).stop 2024-03-16T11:07:36.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 -- 172.21.15.78:0/518243428 shutdown_connections 2024-03-16T11:07:36.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dc7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/518243428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc2850 0x560bc0dc2c70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 -- 172.21.15.78:0/518243428 >> 172.21.15.78:0/518243428 conn(0x560bc0dbddc0 msgr2=0x560bc0dbe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 -- 172.21.15.78:0/518243428 shutdown_connections 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 -- 172.21.15.78:0/518243428 wait complete. 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.221+0000 7fdb11ffc740 1 -- start start 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 1 Processor -- start 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc2850 0x560bc0dccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc3240 0x560bc0dcd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dd4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560bc0dcb300 con 0x560bc0dc2850 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bc0ce3230 con 0x560bc0dc83b0 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560bc0dcb480 con 0x560bc0dc3240 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb11ffc740 10 client.0 _ll_register_callbacks cb 0x560bc0d1f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:36.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb0d8d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dd4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb0c8d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc3240 0x560bc0dcd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb0d8d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dd4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:40950/0 (socket says 172.21.15.78:40950) 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb0c8d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc3240 0x560bc0dcd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57286/0 (socket says 172.21.15.78:57286) 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb0d8d4640 1 -- 172.21.15.78:0/809682840 learned_addr learned my addr 172.21.15.78:0/809682840 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.222+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc2850 0x560bc0dccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/809682840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc3240 msgr2=0x560bc0dcd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc3240 0x560bc0dcd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/809682840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 msgr2=0x560bc0dd4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dd4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/809682840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc0cf2100 con 0x560bc0dc2850 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc3240 0x560bc0dcd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc2850 0x560bc0dccde0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x560bc0dc3cb0 tx=0x7fdaf8012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c3a411b586ae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.223+0000 7fdaf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/809682840 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf800fa40 con 0x560bc0dc2850 2024-03-16T11:07:36.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/809682840 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdaf800fbe0 con 0x560bc0dc2850 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb00003e20 con 0x560bc0dc2850 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/809682840 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdaf800fde0 con 0x560bc0dc2850 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdb11ffc740 10 client.15843 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560bc0a3b3f0 con 0x560bc0dc2850 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/809682840 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb00003e20 con 0x560bc0dc2850 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/809682840 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fdaf8004af0 con 0x560bc0dc2850 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdb11ffc740 10 client.15843 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.224+0000 7fdb11ffc740 10 client.15843 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.225+0000 7fdb11ffc740 20 client.15843 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.225+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0dd8bd0 0x560bc0ddb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.225+0000 7fdb11ffc740 4 client.15843 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.225+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bc0d349d0 con 0x560bc0dd8bd0 2024-03-16T11:07:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.225+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0dd8bd0 0x560bc0ddb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.225+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0dd8bd0 0x560bc0ddb050 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5281db94cce8134d server_cookie=cb7934dfd71b49f4 in_seq=0 out_seq=0 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.225+0000 7fdaf0ff9640 10 client.15843 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.226+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/809682840 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560bc0d349d0 con 0x560bc0dd8bd0 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.226+0000 7fdaf0ff9640 10 client.15843 handle_command_reply: tid=0 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.226+0000 7fdb11ffc740 1 client.15843 shutdown 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.226+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc2850 msgr2=0x560bc0dccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.226+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc2850 0x560bc0dccde0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x560bc0dc3cb0 tx=0x7fdaf8012eb0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 shutdown_connections 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0dd8bd0 0x560bc0ddb050 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdaf0ff9640 0 client.15843 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc83b0 0x560bc0dd4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc3240 0x560bc0dcd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/809682840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc2850 0x560bc0dccde0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 >> 172.21.15.78:0/809682840 conn(0x560bc0dbddc0 msgr2=0x560bc0dbcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 shutdown_connections 2024-03-16T11:07:36.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 1 -- 172.21.15.78:0/809682840 wait complete. 2024-03-16T11:07:36.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.227+0000 7fdb11ffc740 20 client.15843 trim_cache size 0 max 16384 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 Processor -- start 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 -- start start 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dcd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0dbc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb0c8d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb0d0d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dcd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb0c8d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50286/0 (socket says 172.21.15.78:50286) 2024-03-16T11:07:36.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb0d0d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dcd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:40966/0 (socket says 172.21.15.78:40966) 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb0c8d2640 1 -- 172.21.15.78:0/1396371193 learned_addr learned my addr 172.21.15.78:0/1396371193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081ad760 con 0x560bc0dc3240 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081ab030 con 0x560bc0dc2850 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081a8e60 con 0x560bc0dc83b0 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.249+0000 7fdb0d8d4640 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0dbc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdb0c8d2640 1 -- 172.21.15.78:0/1396371193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 msgr2=0x560bc0dbc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0dbc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdb0c8d2640 1 -- 172.21.15.78:0/1396371193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 msgr2=0x560bc0dcd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dcd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdb0c8d2640 1 -- 172.21.15.78:0/1396371193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc0b057d0 con 0x560bc0dc3240 2024-03-16T11:07:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd42e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdb0402c170 tx=0x7fdb040ee490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60cf71d62338d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdaf2ffd640 1 -- 172.21.15.78:0/1396371193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb040c15b0 con 0x560bc0dc3240 2024-03-16T11:07:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdaf2ffd640 1 -- 172.21.15.78:0/1396371193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb040c5c00 con 0x560bc0dc3240 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.250+0000 7fdaf2ffd640 1 -- 172.21.15.78:0/1396371193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04005420 con 0x560bc0dc3240 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 msgr2=0x560bc0dd42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd42e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdb0402c170 tx=0x7fdb040ee490 comp rx=0 tx=0).stop 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 shutdown_connections 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0dbc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd42e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1396371193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dcd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 >> 172.21.15.78:0/1396371193 conn(0x560bc0dbddc0 msgr2=0x560bc0dccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 shutdown_connections 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.251+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1396371193 wait complete. 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.252+0000 7fdb11ffc740 1 -- start start 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.252+0000 7fdb11ffc740 1 Processor -- start 2024-03-16T11:07:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.252+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dd10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.252+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb11ffc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0e3cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dd10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dd10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:40974/0 (socket says 172.21.15.78:40974) 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/1839423025 learned_addr learned my addr 172.21.15.78:0/1839423025 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081a94f0 con 0x560bc0dc3240 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081aa960 con 0x560bc0dc2850 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081a9bc0 con 0x560bc0dc83b0 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb11ffc740 10 client.0 _ll_register_callbacks cb 0x7fdaf8009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d8d4640 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0e3cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/1839423025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 msgr2=0x560bc0e3cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0e3cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/1839423025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 msgr2=0x560bc0dd1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 -- 172.21.15.78:0/1839423025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc0dd1cb0 con 0x560bc0dc2850 2024-03-16T11:07:36.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.253+0000 7fdb0d0d3640 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dd10d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x560bc0dc3cb0 tx=0x7fdaf8012a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bfd13b7383f086 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.254+0000 7fdaf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.254+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/1839423025 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf802c070 con 0x560bc0dc2850 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.254+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/1839423025 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdaf8005220 con 0x560bc0dc2850 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.254+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb040c5c00 con 0x560bc0dc2850 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/1839423025 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdaf8005460 con 0x560bc0dc2850 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 10 client.25679 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560bc0e3e5e0 con 0x560bc0dc2850 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/1839423025 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdaf8005600 con 0x560bc0dc2850 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/1839423025 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fdaf8010800 con 0x560bc0dc2850 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 10 client.25679 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 10 client.25679 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 20 client.25679 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0e3efe0 0x560bc0e41400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 4 client.25679 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fdb040c5c00 con 0x560bc0e3efe0 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.255+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0e3efe0 0x560bc0e41400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdb0c8d2640 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0e3efe0 0x560bc0e41400 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=700c4274a98e8e5a server_cookie=b2b993d2204bf538 in_seq=0 out_seq=0 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdaf0ff9640 10 client.25679 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdaf0ff9640 1 -- 172.21.15.78:0/1839423025 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fdb040c5c00 con 0x560bc0e3efe0 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdaf0ff9640 10 client.25679 handle_command_reply: tid=0 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdb11ffc740 1 client.25679 shutdown 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 msgr2=0x560bc0dd10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dd10d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x560bc0dc3cb0 tx=0x7fdaf8012a20 comp rx=0 tx=0).stop 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 shutdown_connections 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x560bc0e3efe0 0x560bc0e41400 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.256+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc0dc83b0 0x560bc0e3cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.257+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc0dc3240 0x560bc0dd1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.257+0000 7fdb11ffc740 1 --2- 172.21.15.78:0/1839423025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc0dc2850 0x560bc0dd10d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.257+0000 7fdaf0ff9640 0 client.25679 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:07:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.257+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 >> 172.21.15.78:0/1839423025 conn(0x560bc0dbddc0 msgr2=0x560bc0dc18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.257+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 shutdown_connections 2024-03-16T11:07:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.257+0000 7fdb11ffc740 1 -- 172.21.15.78:0/1839423025 wait complete. 2024-03-16T11:07:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.257+0000 7fdb11ffc740 20 client.25679 trim_cache size 0 max 16384 2024-03-16T11:07:36.261 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:07:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdaf40a2900 msgr2=0x7fdaf40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdaf40a2900 0x7fdaf40a4dc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009370 tx=0x7fdb00017000 comp rx=0 tx=0).stop 2024-03-16T11:07:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb08104c30 msgr2=0x7fdb081002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb08104c30 0x7fdb081002b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdb04036040 tx=0x7fdb04005dc0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 shutdown_connections 2024-03-16T11:07:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdaf40a2900 0x7fdaf40a4dc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb08107560 0x7fdb081007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb08104c30 0x7fdb081002b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 --2- 172.21.15.78:0/2239873960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb08102300 0x7fdb08101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.258+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 >> 172.21.15.78:0/2239873960 conn(0x7fdb080fbc70 msgr2=0x7fdb08104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.259+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 shutdown_connections 2024-03-16T11:07:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.259+0000 7fdb0f337640 1 -- 172.21.15.78:0/2239873960 wait complete. 2024-03-16T11:07:36.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T11:07:36.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 -- 172.21.15.78:0/2823945549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf410db00 msgr2=0x7ffaf410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 --2- 172.21.15.78:0/2823945549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf410db00 0x7ffaf410df80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009900 tx=0x7ffae402bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 -- 172.21.15.78:0/2823945549 shutdown_connections 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 --2- 172.21.15.78:0/2823945549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf410db00 0x7ffaf410df80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 --2- 172.21.15.78:0/2823945549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 0x7ffaf410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 --2- 172.21.15.78:0/2823945549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf41087e0 0x7ffaf410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 -- 172.21.15.78:0/2823945549 >> 172.21.15.78:0/2823945549 conn(0x7ffaf406e640 msgr2=0x7ffaf4070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 -- 172.21.15.78:0/2823945549 shutdown_connections 2024-03-16T11:07:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.434+0000 7ffafa834640 1 -- 172.21.15.78:0/2823945549 wait complete. 2024-03-16T11:07:36.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 Processor -- start 2024-03-16T11:07:36.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 -- start start 2024-03-16T11:07:36.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf41087e0 0x7ffaf41a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 0x7ffaf41a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf410db00 0x7ffaf41adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf411c140 con 0x7ffaf410db00 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf411bfc0 con 0x7ffaf410b150 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.435+0000 7ffafa834640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf411c2c0 con 0x7ffaf41087e0 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 0x7ffaf41a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf41087e0 0x7ffaf41a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 0x7ffaf41a6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:40992/0 (socket says 172.21.15.78:40992) 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 -- 172.21.15.78:0/4036717494 learned_addr learned my addr 172.21.15.78:0/4036717494 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf8dd1640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf410db00 0x7ffaf41adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 -- 172.21.15.78:0/4036717494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf41087e0 msgr2=0x7ffaf41a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf41087e0 0x7ffaf41a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 -- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf410db00 msgr2=0x7ffaf41adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf410db00 0x7ffaf41adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 -- 172.21.15.78:0/4036717494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf41ae450 con 0x7ffaf410b150 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf41087e0 0x7ffaf41a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.436+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 0x7ffaf41a6dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffae800d950 tx=0x7ffae800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.437+0000 7ffaf17fa640 1 -- 172.21.15.78:0/4036717494 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae8017030 con 0x7ffaf410b150 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.437+0000 7ffaf17fa640 1 -- 172.21.15.78:0/4036717494 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffae8004020 con 0x7ffaf410b150 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.437+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf41ae740 con 0x7ffaf410b150 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.437+0000 7ffaf17fa640 1 -- 172.21.15.78:0/4036717494 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae80047a0 con 0x7ffaf410b150 2024-03-16T11:07:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.437+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf41aece0 con 0x7ffaf410b150 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffaf17fa640 1 -- 172.21.15.78:0/4036717494 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ffae80041d0 con 0x7ffaf410b150 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffafd41b740 1 Processor -- start 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffafd41b740 1 -- start start 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e38231ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e38231fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e382322840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffafd41b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e38227f490 con 0x55e38231b240 2024-03-16T11:07:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.440+0000 7ffafd41b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e382227c90 con 0x55e38231a850 2024-03-16T11:07:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffafd41b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e38227b530 con 0x55e3823203b0 2024-03-16T11:07:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e38231fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e38231fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50316/0 (socket says 172.21.15.78:50316) 2024-03-16T11:07:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 -- 172.21.15.78:0/1793607120 learned_addr learned my addr 172.21.15.78:0/1793607120 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf8dd1640 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e38231ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 -- 172.21.15.78:0/1793607120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 msgr2=0x55e382322840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T11:07:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e382322840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 -- 172.21.15.78:0/1793607120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 msgr2=0x55e38231ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e38231ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 -- 172.21.15.78:0/1793607120 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3822e2500 con 0x55e38231b240 2024-03-16T11:07:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf8dd1640 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e38231ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf17fa640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffadc0a2880 0x7ffadc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.441+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e38231fd40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffaec011860 tx=0x7ffaec011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0dd209c479e4559 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.442+0000 7ffada7fc640 1 -- 172.21.15.78:0/1793607120 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec0112f0 con 0x55e38231b240 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.442+0000 7ffada7fc640 1 -- 172.21.15.78:0/1793607120 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffaec00f9a0 con 0x55e38231b240 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.442+0000 7ffada7fc640 1 -- 172.21.15.78:0/1793607120 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec005630 con 0x55e38231b240 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.442+0000 7ffaf17fa640 1 -- 172.21.15.78:0/4036717494 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ffae80ce400 con 0x7ffaf410b150 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.442+0000 7ffaf8dd1640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffadc0a2880 0x7ffadc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 -- 172.21.15.78:0/1793607120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 msgr2=0x55e38231fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e38231fd40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffaec011860 tx=0x7ffaec011d20 comp rx=0 tx=0).stop 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffaf8dd1640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffadc0a2880 0x7ffadc0a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffae4005ab0 tx=0x7ffae402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 -- 172.21.15.78:0/1793607120 shutdown_connections 2024-03-16T11:07:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e382322840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e38231fd40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1793607120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e38231ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 -- 172.21.15.78:0/1793607120 >> 172.21.15.78:0/1793607120 conn(0x55e382315dc0 msgr2=0x55e3823161c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 -- 172.21.15.78:0/1793607120 shutdown_connections 2024-03-16T11:07:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.443+0000 7ffafd41b740 1 -- 172.21.15.78:0/1793607120 wait complete. 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.444+0000 7ffafd41b740 1 -- start start 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.444+0000 7ffafd41b740 1 Processor -- start 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e38231a850 0x55e382324d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e3823252b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3823203b0 0x55e38232c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e38231a850 0x55e382324d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e38231a850 0x55e382324d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57352/0 (socket says 172.21.15.78:57352) 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffaf3fff640 1 -- 172.21.15.78:0/2620855863 learned_addr learned my addr 172.21.15.78:0/2620855863 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e3823252b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffafd41b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e382323360 con 0x55e38231b240 2024-03-16T11:07:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e38223b230 con 0x55e3823203b0 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e3823234e0 con 0x55e38231a850 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffafd41b740 10 client.0 _ll_register_callbacks cb 0x55e382277010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.445+0000 7ffaf8dd1640 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3823203b0 0x55e38232c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffaf37fe640 1 -- 172.21.15.78:0/2620855863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e38231a850 msgr2=0x55e382324d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e38231a850 0x55e382324d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffaf37fe640 1 -- 172.21.15.78:0/2620855863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3823203b0 msgr2=0x55e38232c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3823203b0 0x55e38232c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffaf37fe640 1 -- 172.21.15.78:0/2620855863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e38224a100 con 0x55e38231b240 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e3823252b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffae80cbed0 tx=0x7ffae809ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d554b06a00dab5c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffad37fe640 1 -- 172.21.15.78:0/2620855863 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae80b3e60 con 0x55e38231b240 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffad37fe640 1 -- 172.21.15.78:0/2620855863 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ffae809a1f0 con 0x55e38231b240 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.446+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaec00f9a0 con 0x55e38231b240 2024-03-16T11:07:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffad37fe640 1 -- 172.21.15.78:0/2620855863 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ffae809e070 con 0x55e38231b240 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 10 client.15861 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e381f933f0 con 0x55e38231b240 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffad37fe640 1 -- 172.21.15.78:0/2620855863 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffaec00f9a0 con 0x55e38231b240 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffad37fe640 1 -- 172.21.15.78:0/2620855863 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ffae8009b50 con 0x55e38231b240 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 10 client.15861 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 10 client.15861 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 20 client.15861 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e3823334a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 4 client.15861 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e38228c9d0 con 0x55e382331090 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.447+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e3823334a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.448+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e3823334a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=542f8f142bf7f4bb server_cookie=b15b0fabdd5a4ed5 in_seq=0 out_seq=0 2024-03-16T11:07:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.448+0000 7ffad37fe640 10 client.15861 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffad37fe640 1 -- 172.21.15.78:0/2620855863 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e38228c9d0 con 0x55e382331090 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffad37fe640 10 client.15861 handle_command_reply: tid=0 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffafd41b740 1 client.15861 shutdown 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 msgr2=0x55e3823252b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffafd41b740 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e3823252b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffae80cbed0 tx=0x7ffae809ac80 comp rx=0 tx=0).stop 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 shutdown_connections 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffafd41b740 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e3823334a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.449+0000 7ffad37fe640 0 client.15861 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.450+0000 7ffafd41b740 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3823203b0 0x55e38232c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.450+0000 7ffafd41b740 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e3823252b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.450+0000 7ffafd41b740 1 --2- 172.21.15.78:0/2620855863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e38231a850 0x55e382324d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.450+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 >> 172.21.15.78:0/2620855863 conn(0x55e382315dc0 msgr2=0x55e382314fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.450+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 shutdown_connections 2024-03-16T11:07:36.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.450+0000 7ffafd41b740 1 -- 172.21.15.78:0/2620855863 wait complete. 2024-03-16T11:07:36.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.450+0000 7ffafd41b740 20 client.15861 trim_cache size 0 max 16384 2024-03-16T11:07:36.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:36 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4251797894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:07:36.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.471+0000 7ffafd41b740 1 Processor -- start 2024-03-16T11:07:36.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.471+0000 7ffafd41b740 1 -- start start 2024-03-16T11:07:36.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231a850 0x55e38231c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231b240 0x55e38231cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffafd41b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf41ba4c0 con 0x55e38231a850 2024-03-16T11:07:36.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffafd41b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf41b7d90 con 0x55e38231b240 2024-03-16T11:07:36.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffafd41b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf41b5bc0 con 0x55e3823203b0 2024-03-16T11:07:36.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231a850 0x55e38231c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231b240 0x55e38231cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231a850 0x55e38231c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50340/0 (socket says 172.21.15.78:50340) 2024-03-16T11:07:36.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 -- 172.21.15.78:0/3268557942 learned_addr learned my addr 172.21.15.78:0/3268557942 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 -- 172.21.15.78:0/3268557942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 msgr2=0x55e38232c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 -- 172.21.15.78:0/3268557942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231b240 msgr2=0x55e38231cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231b240 0x55e38231cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf3fff640 1 -- 172.21.15.78:0/3268557942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e38205d7d0 con 0x55e38231a850 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.472+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231b240 0x55e38231cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.473+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231a850 0x55e38231c7d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009b90 tx=0x7ffaec000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f36e9591f7e7c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.473+0000 7ffad9ffb640 1 -- 172.21.15.78:0/3268557942 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec01e040 con 0x55e38231a850 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.473+0000 7ffad9ffb640 1 -- 172.21.15.78:0/3268557942 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffaec01f040 con 0x55e38231a850 2024-03-16T11:07:36.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.473+0000 7ffad9ffb640 1 -- 172.21.15.78:0/3268557942 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec005370 con 0x55e38231a850 2024-03-16T11:07:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 -- 172.21.15.78:0/3268557942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231a850 msgr2=0x55e38231c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231a850 0x55e38231c7d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009b90 tx=0x7ffaec000f40 comp rx=0 tx=0).stop 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 -- 172.21.15.78:0/3268557942 shutdown_connections 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231b240 0x55e38231cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 --2- 172.21.15.78:0/3268557942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231a850 0x55e38231c7d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 -- 172.21.15.78:0/3268557942 >> 172.21.15.78:0/3268557942 conn(0x55e382315dc0 msgr2=0x55e3823161a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 -- 172.21.15.78:0/3268557942 shutdown_connections 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.474+0000 7ffafd41b740 1 -- 172.21.15.78:0/3268557942 wait complete. 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 -- start start 2024-03-16T11:07:36.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 Processor -- start 2024-03-16T11:07:36.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e382329450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e382329990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf41b6250 con 0x55e38231b240 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf41b76c0 con 0x55e38231a850 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf41b6920 con 0x55e3823203b0 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffafd41b740 10 client.0 _ll_register_callbacks cb 0x7ffae809a700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e382329450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffaf8dd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffaf37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e382329990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffaf8dd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57374/0 (socket says 172.21.15.78:57374) 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.475+0000 7ffaf8dd1640 1 -- 172.21.15.78:0/1264454424 learned_addr learned my addr 172.21.15.78:0/1264454424 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffaf37fe640 1 -- 172.21.15.78:0/1264454424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 msgr2=0x55e38232a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffaf37fe640 1 -- 172.21.15.78:0/1264454424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 msgr2=0x55e382329450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e382329450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffaf37fe640 1 -- 172.21.15.78:0/1264454424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e38232aed0 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffaf37fe640 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e382329990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ffae8099050 tx=0x7ffae8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2df0f58596f157 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffad37fe640 1 -- 172.21.15.78:0/1264454424 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae8081020 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffad37fe640 1 -- 172.21.15.78:0/1264454424 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ffae8009a30 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.476+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaec01f040 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffad37fe640 1 -- 172.21.15.78:0/1264454424 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ffae80b3e60 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 10 client.15867 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e38232c1b0 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffad37fe640 1 -- 172.21.15.78:0/1264454424 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffae80b0030 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffad37fe640 1 -- 172.21.15.78:0/1264454424 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ffae809fe10 con 0x55e38231b240 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 10 client.15867 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 10 client.15867 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 20 client.15867 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e382399590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 4 client.15867 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:07:36.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ffaec01f040 con 0x55e382331090 2024-03-16T11:07:36.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.477+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e382399590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.478+0000 7ffaf3fff640 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e382399590 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bfb5e5d0ff313340 server_cookie=18d57a446b484a6 in_seq=0 out_seq=0 2024-03-16T11:07:36.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.478+0000 7ffad37fe640 10 client.15867 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.478+0000 7ffad37fe640 1 -- 172.21.15.78:0/1264454424 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ffaec01f040 con 0x55e382331090 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffad37fe640 10 client.15867 handle_command_reply: tid=0 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 client.15867 shutdown 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 msgr2=0x55e382329990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e382329990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ffae8099050 tx=0x7ffae8000bc0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 shutdown_connections 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x55e382331090 0x55e382399590 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e3823203b0 0x55e38232a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e38231b240 0x55e382329990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 --2- 172.21.15.78:0/1264454424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e38231a850 0x55e382329450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffad37fe640 0 client.15867 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:07:36.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 >> 172.21.15.78:0/1264454424 conn(0x55e382315dc0 msgr2=0x55e38231c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 shutdown_connections 2024-03-16T11:07:36.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 1 -- 172.21.15.78:0/1264454424 wait complete. 2024-03-16T11:07:36.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.479+0000 7ffafd41b740 20 client.15867 trim_cache size 0 max 16384 2024-03-16T11:07:36.483 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffadc0a2880 msgr2=0x7ffadc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffadc0a2880 0x7ffadc0a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffae4005ab0 tx=0x7ffae402c7e0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 msgr2=0x7ffaf41a6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 0x7ffaf41a6dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffae800d950 tx=0x7ffae800de10 comp rx=0 tx=0).stop 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 shutdown_connections 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf410db00 0x7ffaf41adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaf410b150 0x7ffaf41a6dc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ffadc0a2880 0x7ffadc0a4d40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 --2- 172.21.15.78:0/4036717494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf41087e0 0x7ffaf41a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 >> 172.21.15.78:0/4036717494 conn(0x7ffaf406e640 msgr2=0x7ffaf410b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 shutdown_connections 2024-03-16T11:07:36.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.480+0000 7ffafa834640 1 -- 172.21.15.78:0/4036717494 wait complete. 2024-03-16T11:07:36.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:36 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4251797894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:07:36.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T11:07:36.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:36 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4251797894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:07:36.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 -- 172.21.15.78:0/1417480330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 msgr2=0x7f62a8110790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 --2- 172.21.15.78:0/1417480330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 0x7f62a8110790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6298009900 tx=0x7f629802bc30 comp rx=0 tx=0).stop 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 -- 172.21.15.78:0/1417480330 shutdown_connections 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 --2- 172.21.15.78:0/1417480330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 0x7f62a8110790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 --2- 172.21.15.78:0/1417480330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a810b9a0 0x7f62a810dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 --2- 172.21.15.78:0/1417480330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a8109070 0x7f62a810b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 -- 172.21.15.78:0/1417480330 >> 172.21.15.78:0/1417480330 conn(0x7f62a806e9e0 msgr2=0x7f62a8070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 -- 172.21.15.78:0/1417480330 shutdown_connections 2024-03-16T11:07:36.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.652+0000 7f62b0187640 1 -- 172.21.15.78:0/1417480330 wait complete. 2024-03-16T11:07:36.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 Processor -- start 2024-03-16T11:07:36.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 -- start start 2024-03-16T11:07:36.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a8109070 0x7f62a810b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a810b9a0 0x7f62a8109670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 0x7f62a8109bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8113910 con 0x7f62a810b9a0 2024-03-16T11:07:36.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8113790 con 0x7f62a810e2d0 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62b0187640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8113a90 con 0x7f62a8109070 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62ad722640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a810b9a0 0x7f62a8109670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62ad722640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a810b9a0 0x7f62a8109670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50374/0 (socket says 172.21.15.78:50374) 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62adf23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a8109070 0x7f62a810b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62ae724640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 0x7f62a8109bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.653+0000 7f62ad722640 1 -- 172.21.15.78:0/882108776 learned_addr learned my addr 172.21.15.78:0/882108776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62adf23640 1 -- 172.21.15.78:0/882108776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 msgr2=0x7f62a8109bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62adf23640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 0x7f62a8109bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62adf23640 1 -- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a810b9a0 msgr2=0x7f62a8109670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62adf23640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a810b9a0 0x7f62a8109670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62adf23640 1 -- 172.21.15.78:0/882108776 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a810a0f0 con 0x7f62a8109070 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62adf23640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a8109070 0x7f62a810b030 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009fd0 tx=0x7f62a000ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f629effd640 1 -- 172.21.15.78:0/882108776 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0004b20 con 0x7f62a8109070 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a810a3e0 con 0x7f62a8109070 2024-03-16T11:07:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a81aece0 con 0x7f62a8109070 2024-03-16T11:07:36.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f629effd640 1 -- 172.21.15.78:0/882108776 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a0004cc0 con 0x7f62a8109070 2024-03-16T11:07:36.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.654+0000 7f629effd640 1 -- 172.21.15.78:0/882108776 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a000f620 con 0x7f62a8109070 2024-03-16T11:07:36.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.657+0000 7f629effd640 1 -- 172.21.15.78:0/882108776 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f62a000f7c0 con 0x7f62a8109070 2024-03-16T11:07:36.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.657+0000 7f62b2d75740 1 Processor -- start 2024-03-16T11:07:36.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62b2d75740 1 -- start start 2024-03-16T11:07:36.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e04c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e09d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e0a3b0 0x55dd15e0c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62b2d75740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd15d69490 con 0x55dd15e05240 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62b2d75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd15d11c90 con 0x55dd15e04850 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62b2d75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd15d65530 con 0x55dd15e0a3b0 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e04c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62adf23640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e09d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e04c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41048/0 (socket says 172.21.15.78:41048) 2024-03-16T11:07:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 -- 172.21.15.78:0/674896034 learned_addr learned my addr 172.21.15.78:0/674896034 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ad722640 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e0a3b0 0x55dd15e0c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f629effd640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62940a2880 0x7f62940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 -- 172.21.15.78:0/674896034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e0a3b0 msgr2=0x55dd15e0c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e0a3b0 0x55dd15e0c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 -- 172.21.15.78:0/674896034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 msgr2=0x55dd15e09d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e09d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.658+0000 7f62ae724640 1 -- 172.21.15.78:0/674896034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd15dcc500 con 0x55dd15e04850 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.659+0000 7f62adf23640 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e09d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:36.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.659+0000 7f629effd640 1 -- 172.21.15.78:0/882108776 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f62a00ced00 con 0x7f62a8109070 2024-03-16T11:07:36.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.659+0000 7f62ae724640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62940a2880 0x7f62940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.659+0000 7f62ae724640 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e04c70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6298009900 tx=0x7f62980095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a64728c47a7c14f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.659+0000 7f6293fff640 1 -- 172.21.15.78:0/674896034 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629802cc80 con 0x55dd15e04850 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.659+0000 7f6293fff640 1 -- 172.21.15.78:0/674896034 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62980055b0 con 0x55dd15e04850 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.659+0000 7f6293fff640 1 -- 172.21.15.78:0/674896034 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62980035f0 con 0x55dd15e04850 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 -- 172.21.15.78:0/674896034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 msgr2=0x55dd15e04c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e04c70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6298009900 tx=0x7f62980095e0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62ae724640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62940a2880 0x7f62940a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f629804a000 tx=0x7f629804d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 -- 172.21.15.78:0/674896034 shutdown_connections 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e0a3b0 0x55dd15e0c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e09d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 --2- 172.21.15.78:0/674896034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e04c70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 -- 172.21.15.78:0/674896034 >> 172.21.15.78:0/674896034 conn(0x55dd15dffdc0 msgr2=0x55dd15e001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 -- 172.21.15.78:0/674896034 shutdown_connections 2024-03-16T11:07:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.660+0000 7f62b2d75740 1 -- 172.21.15.78:0/674896034 wait complete. 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 -- start start 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 Processor -- start 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e0ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e0a3b0 0x55dd15e16300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd15e0d3b0 con 0x55dd15e05240 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd15d25230 con 0x55dd15e0a3b0 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62b2d75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd15e0d530 con 0x55dd15e04850 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.661+0000 7f62adf23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e0ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e0ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57394/0 (socket says 172.21.15.78:57394) 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62b2d75740 10 client.0 _ll_register_callbacks cb 0x55dd15d61010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 -- 172.21.15.78:0/946382070 learned_addr learned my addr 172.21.15.78:0/946382070 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62ae724640 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e0a3b0 0x55dd15e16300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62ad722640 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 -- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e0a3b0 msgr2=0x55dd15e16300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e0a3b0 0x55dd15e16300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 -- 172.21.15.78:0/946382070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 msgr2=0x55dd15e0f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e0f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 -- 172.21.15.78:0/946382070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd15d34100 con 0x55dd15e04850 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62adf23640 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e0ed80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c760 tx=0x7f62a00d24e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db09e2f660032fec server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62917fa640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62917fa640 1 -- 172.21.15.78:0/946382070 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a009b000 con 0x55dd15e04850 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62917fa640 1 -- 172.21.15.78:0/946382070 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f62a009acf0 con 0x55dd15e04850 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.662+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62980055b0 con 0x55dd15e04850 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62917fa640 1 -- 172.21.15.78:0/946382070 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f62a00860f0 con 0x55dd15e04850 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62b2d75740 10 client.25690 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55dd15a7d3f0 con 0x55dd15e04850 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62917fa640 1 -- 172.21.15.78:0/946382070 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a009bda0 con 0x55dd15e04850 2024-03-16T11:07:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62917fa640 1 -- 172.21.15.78:0/946382070 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f62a001f020 con 0x55dd15e04850 2024-03-16T11:07:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62b2d75740 10 client.25690 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62b2d75740 10 client.25690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.663+0000 7f62b2d75740 20 client.25690 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.664+0000 7f62b2d75740 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e1acb0 0x55dd15e1d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.664+0000 7f62b2d75740 4 client.25690 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.664+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f62980055b0 con 0x55dd15e1acb0 2024-03-16T11:07:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.664+0000 7f62ad722640 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e1acb0 0x55dd15e1d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.664+0000 7f62ad722640 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e1acb0 0x55dd15e1d130 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=481fdd75278a3366 server_cookie=595db57848f48099 in_seq=0 out_seq=0 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.664+0000 7f62917fa640 10 client.25690 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.665+0000 7f62917fa640 1 -- 172.21.15.78:0/946382070 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f62980055b0 con 0x55dd15e1acb0 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.665+0000 7f62917fa640 10 client.25690 handle_command_reply: tid=0 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.665+0000 7f62b2d75740 1 client.25690 shutdown 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 msgr2=0x55dd15e0ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e0ed80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c760 tx=0x7f62a00d24e0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 shutdown_connections 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e1acb0 0x55dd15e1d130 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e0a3b0 0x55dd15e16300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e05240 0x55dd15e0f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62917fa640 0 client.25690 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 --2- 172.21.15.78:0/946382070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e0ed80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 >> 172.21.15.78:0/946382070 conn(0x55dd15dffdc0 msgr2=0x55dd15dfefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 shutdown_connections 2024-03-16T11:07:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 1 -- 172.21.15.78:0/946382070 wait complete. 2024-03-16T11:07:36.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.666+0000 7f62b2d75740 20 client.25690 trim_cache size 0 max 16384 2024-03-16T11:07:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.687+0000 7f62b2d75740 1 Processor -- start 2024-03-16T11:07:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.687+0000 7f62b2d75740 1 -- start start 2024-03-16T11:07:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.687+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.687+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e05240 0x55dd15e16290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.687+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e06be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57404/0 (socket says 172.21.15.78:57404) 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62ad722640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e05240 0x55dd15e16290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62ae724640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e06be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62ae724640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e06be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50406/0 (socket says 172.21.15.78:50406) 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62ad722640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e05240 0x55dd15e16290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41062/0 (socket says 172.21.15.78:41062) 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 -- 172.21.15.78:0/785762707 learned_addr learned my addr 172.21.15.78:0/785762707 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f628c010d40 con 0x55dd15e0a3b0 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f628c00e610 con 0x55dd15e05240 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f628c00c440 con 0x55dd15e04850 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 -- 172.21.15.78:0/785762707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e05240 msgr2=0x55dd15e16290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e05240 0x55dd15e16290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 -- 172.21.15.78:0/785762707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 msgr2=0x55dd15e06be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e06be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62adf23640 1 -- 172.21.15.78:0/785762707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd15b477d0 con 0x55dd15e04850 2024-03-16T11:07:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.688+0000 7f62ae724640 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e06be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:36.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.689+0000 7f62adf23640 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e066a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f62a0098020 tx=0x7f62a00d2360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ede63fc99736d64 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.689+0000 7f62937fe640 1 -- 172.21.15.78:0/785762707 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a009bda0 con 0x55dd15e04850 2024-03-16T11:07:36.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.689+0000 7f62937fe640 1 -- 172.21.15.78:0/785762707 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a009b6a0 con 0x55dd15e04850 2024-03-16T11:07:36.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.689+0000 7f62937fe640 1 -- 172.21.15.78:0/785762707 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0097650 con 0x55dd15e04850 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 msgr2=0x55dd15e066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e066a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f62a0098020 tx=0x7f62a00d2360 comp rx=0 tx=0).stop 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 shutdown_connections 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e06be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e05240 0x55dd15e16290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 --2- 172.21.15.78:0/785762707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e04850 0x55dd15e066a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 >> 172.21.15.78:0/785762707 conn(0x55dd15dffdc0 msgr2=0x55dd15e001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 shutdown_connections 2024-03-16T11:07:36.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 -- 172.21.15.78:0/785762707 wait complete. 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 -- start start 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.690+0000 7f62b2d75740 1 Processor -- start 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e13790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e05240 0x55dd15e13cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62b2d75740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e14c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62b2d75740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f628c00cad0 con 0x55dd15e0a3b0 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62b2d75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f628c00df40 con 0x55dd15e04850 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62b2d75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f628c00d1a0 con 0x55dd15e05240 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62b2d75740 10 client.0 _ll_register_callbacks cb 0x7f62a0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e13790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62ad722640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e05240 0x55dd15e13cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e13790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41074/0 (socket says 172.21.15.78:41074) 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62ad722640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e05240 0x55dd15e13cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57410/0 (socket says 172.21.15.78:57410) 2024-03-16T11:07:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 -- 172.21.15.78:0/453455011 learned_addr learned my addr 172.21.15.78:0/453455011 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62ae724640 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e14c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 -- 172.21.15.78:0/453455011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e05240 msgr2=0x55dd15e13cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e05240 0x55dd15e13cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 -- 172.21.15.78:0/453455011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 msgr2=0x55dd15e14c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e14c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.691+0000 7f62adf23640 1 -- 172.21.15.78:0/453455011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd15e15260 con 0x55dd15e04850 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.692+0000 7f62adf23640 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e13790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62a0099950 tx=0x7f62a00976a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861301597b318571 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.692+0000 7f62917fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.692+0000 7f62917fa640 1 -- 172.21.15.78:0/453455011 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a009bda0 con 0x55dd15e04850 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.692+0000 7f62917fa640 1 -- 172.21.15.78:0/453455011 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f62a00d2410 con 0x55dd15e04850 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.692+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a009b6a0 con 0x55dd15e04850 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62917fa640 1 -- 172.21.15.78:0/453455011 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f62a00a0920 con 0x55dd15e04850 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 10 client.25715 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55dd15e15260 con 0x55dd15e04850 2024-03-16T11:07:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62917fa640 1 -- 172.21.15.78:0/453455011 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a00a0ac0 con 0x55dd15e04850 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62917fa640 1 -- 172.21.15.78:0/453455011 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f62a001f070 con 0x55dd15e04850 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 10 client.25715 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 10 client.25715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 20 client.25715 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e813b0 0x55dd15e837a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 4 client.25715 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.693+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f62a009b6a0 con 0x55dd15e813b0 2024-03-16T11:07:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.694+0000 7f62ad722640 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e813b0 0x55dd15e837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:36.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.694+0000 7f62ad722640 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e813b0 0x55dd15e837a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2529501461ec0b32 server_cookie=247527987d212447 in_seq=0 out_seq=0 2024-03-16T11:07:36.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.694+0000 7f62917fa640 10 client.25715 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:36.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.694+0000 7f62917fa640 1 -- 172.21.15.78:0/453455011 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f62a009b6a0 con 0x55dd15e813b0 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.694+0000 7f62917fa640 10 client.25715 handle_command_reply: tid=0 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.694+0000 7f62b2d75740 1 client.25715 shutdown 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 msgr2=0x55dd15e13790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e13790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62a0099950 tx=0x7f62a00976a0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 shutdown_connections 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dd15e813b0 0x55dd15e837a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd15e0a3b0 0x55dd15e14c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd15e05240 0x55dd15e13cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 --2- 172.21.15.78:0/453455011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd15e04850 0x55dd15e13790 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 >> 172.21.15.78:0/453455011 conn(0x55dd15dffdc0 msgr2=0x55dd15e0eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62917fa640 0 client.25715 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 shutdown_connections 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 1 -- 172.21.15.78:0/453455011 wait complete. 2024-03-16T11:07:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.695+0000 7f62b2d75740 20 client.25715 trim_cache size 0 max 16384 2024-03-16T11:07:36.700 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:07:36.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62940a2880 msgr2=0x7f62940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62940a2880 0x7f62940a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f629804a000 tx=0x7f629804d000 comp rx=0 tx=0).stop 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a8109070 msgr2=0x7f62a810b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a8109070 0x7f62a810b030 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009fd0 tx=0x7f62a000ef90 comp rx=0 tx=0).stop 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 shutdown_connections 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a810e2d0 0x7f62a8109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a810b9a0 0x7f62a8109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62940a2880 0x7f62940a4d40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 --2- 172.21.15.78:0/882108776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a8109070 0x7f62a810b030 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.696+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 >> 172.21.15.78:0/882108776 conn(0x7f62a806e9e0 msgr2=0x7f62a8070dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.697+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 shutdown_connections 2024-03-16T11:07:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:36.697+0000 7f62b0187640 1 -- 172.21.15.78:0/882108776 wait complete. 2024-03-16T11:07:37.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:37 smithi098 ceph-mon[35848]: pgmap v397: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 MiB/s wr, 1.70k op/s 2024-03-16T11:07:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:37 smithi153 ceph-mon[35521]: pgmap v397: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 MiB/s wr, 1.70k op/s 2024-03-16T11:07:37.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:37 smithi078 ceph-mon[32235]: pgmap v397: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 MiB/s wr, 1.70k op/s 2024-03-16T11:07:37.713 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T11:07:37.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:07:37.713 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 -- 172.21.15.78:0/402080694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6501034f0 msgr2=0x7fe650109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 --2- 172.21.15.78:0/402080694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6501034f0 0x7fe650109d80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe648009ca0 tx=0x7fe64802c140 comp rx=0 tx=0).stop 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 -- 172.21.15.78:0/402080694 shutdown_connections 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 --2- 172.21.15.78:0/402080694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6501034f0 0x7fe650109d80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 --2- 172.21.15.78:0/402080694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe650102b30 0x7fe650102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 --2- 172.21.15.78:0/402080694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500ffac0 0x7fe6500ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 -- 172.21.15.78:0/402080694 >> 172.21.15.78:0/402080694 conn(0x7fe6500fb3d0 msgr2=0x7fe6500fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 -- 172.21.15.78:0/402080694 shutdown_connections 2024-03-16T11:07:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.868+0000 7fe656427640 1 -- 172.21.15.78:0/402080694 wait complete. 2024-03-16T11:07:37.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.869+0000 7fe656427640 1 Processor -- start 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.869+0000 7fe656427640 1 -- start start 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe656427640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500ffac0 0x7fe65019de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe656427640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe650102b30 0x7fe65019e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe656427640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6501034f0 0x7fe6501a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe656427640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010d350 con 0x7fe650102b30 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe656427640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010d1d0 con 0x7fe6500ffac0 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe656427640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010d4d0 con 0x7fe6501034f0 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe650102b30 0x7fe65019e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500ffac0 0x7fe65019de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe650102b30 0x7fe65019e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50448/0 (socket says 172.21.15.78:50448) 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64f7fe640 1 -- 172.21.15.78:0/3875436253 learned_addr learned my addr 172.21.15.78:0/3875436253 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe6549c4640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6501034f0 0x7fe6501a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64ffff640 1 -- 172.21.15.78:0/3875436253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6501034f0 msgr2=0x7fe6501a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64ffff640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6501034f0 0x7fe6501a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64ffff640 1 -- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe650102b30 msgr2=0x7fe65019e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64ffff640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe650102b30 0x7fe65019e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64ffff640 1 -- 172.21.15.78:0/3875436253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6501a5a30 con 0x7fe6500ffac0 2024-03-16T11:07:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.870+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe650102b30 0x7fe65019e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.871+0000 7fe64ffff640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500ffac0 0x7fe65019de10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe64000e9c0 tx=0x7fe64000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.871+0000 7fe64d7fa640 1 -- 172.21.15.78:0/3875436253 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64000cd80 con 0x7fe6500ffac0 2024-03-16T11:07:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.871+0000 7fe64d7fa640 1 -- 172.21.15.78:0/3875436253 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6400108e0 con 0x7fe6500ffac0 2024-03-16T11:07:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.871+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6501a5d20 con 0x7fe6500ffac0 2024-03-16T11:07:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.871+0000 7fe64d7fa640 1 -- 172.21.15.78:0/3875436253 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640018860 con 0x7fe6500ffac0 2024-03-16T11:07:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.871+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6501a61b0 con 0x7fe6500ffac0 2024-03-16T11:07:37.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.874+0000 7fe64d7fa640 1 -- 172.21.15.78:0/3875436253 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe64001f020 con 0x7fe6500ffac0 2024-03-16T11:07:37.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 Processor -- start 2024-03-16T11:07:37.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 -- start start 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e1edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e23e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e26980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55cd54d830d0 con 0x55cd54e244f0 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cd54d2bc70 con 0x55cd54e1e990 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe659012740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cd54d7f170 con 0x55cd54e1f380 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe64f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e26980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.875+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e1edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.876+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e1edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41106/0 (socket says 172.21.15.78:41106) 2024-03-16T11:07:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.876+0000 7fe6549c4640 1 -- 172.21.15.78:0/2848371748 learned_addr learned my addr 172.21.15.78:0/2848371748 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.876+0000 7fe64ffff640 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e23e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.876+0000 7fe6549c4640 1 -- 172.21.15.78:0/2848371748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 msgr2=0x55cd54e23e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.876+0000 7fe6549c4640 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e23e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.876+0000 7fe6549c4640 1 -- 172.21.15.78:0/2848371748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 msgr2=0x55cd54e26980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.876+0000 7fe64d7fa640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe6380a2900 0x7fe6380a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe6549c4640 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e26980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe6549c4640 1 -- 172.21.15.78:0/2848371748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd54b5f7e0 con 0x55cd54e1e990 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e26980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe64ffff640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe6380a2900 0x7fe6380a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe64d7fa640 1 -- 172.21.15.78:0/3875436253 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe6400ce9e0 con 0x7fe6500ffac0 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe6549c4640 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e1edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe648037040 tx=0x7fe6480093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31ffab7648a5ea57 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe64ffff640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe6380a2900 0x7fe6380a4dc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6400a44b0 tx=0x7fe6400d4530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:37.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe6367fc640 1 -- 172.21.15.78:0/2848371748 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648043070 con 0x55cd54e1e990 2024-03-16T11:07:37.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe6367fc640 1 -- 172.21.15.78:0/2848371748 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe648032df0 con 0x55cd54e1e990 2024-03-16T11:07:37.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.877+0000 7fe6367fc640 1 -- 172.21.15.78:0/2848371748 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648039040 con 0x55cd54e1e990 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 -- 172.21.15.78:0/2848371748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 msgr2=0x55cd54e1edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e1edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe648037040 tx=0x7fe6480093f0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 -- 172.21.15.78:0/2848371748 shutdown_connections 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e26980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e23e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 --2- 172.21.15.78:0/2848371748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e1edb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 -- 172.21.15.78:0/2848371748 >> 172.21.15.78:0/2848371748 conn(0x55cd54de6c50 msgr2=0x55cd54e1a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 -- 172.21.15.78:0/2848371748 shutdown_connections 2024-03-16T11:07:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 -- 172.21.15.78:0/2848371748 wait complete. 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.878+0000 7fe659012740 1 -- start start 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe659012740 1 Processor -- start 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe659012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e28e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe659012740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e293b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe659012740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e303f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e28e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e28e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41114/0 (socket says 172.21.15.78:41114) 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe6549c4640 1 -- 172.21.15.78:0/620804966 learned_addr learned my addr 172.21.15.78:0/620804966 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55cd54e27440 con 0x55cd54e244f0 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cd54d3f0d0 con 0x55cd54e1e990 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cd54e275c0 con 0x55cd54e1f380 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe6549c4640 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e303f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.879+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e293b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe6549c4640 1 -- 172.21.15.78:0/620804966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 msgr2=0x55cd54e293b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe6549c4640 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e293b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe6549c4640 1 -- 172.21.15.78:0/620804966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 msgr2=0x55cd54e28e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe6549c4640 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e28e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe6549c4640 1 -- 172.21.15.78:0/620804966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd54bbff70 con 0x55cd54e244f0 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe659012740 10 client.0 _ll_register_callbacks cb 0x55cd54d08760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe648032df0 con 0x55cd54e244f0 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe6549c4640 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e303f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe648006010 tx=0x7fe648005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6981534cad03ee14 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe62f7fe640 10 client.15897 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe62f7fe640 1 -- 172.21.15.78:0/620804966 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648059050 con 0x55cd54e244f0 2024-03-16T11:07:37.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe62f7fe640 1 -- 172.21.15.78:0/620804966 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe6480043b0 con 0x55cd54e244f0 2024-03-16T11:07:37.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe62f7fe640 1 -- 172.21.15.78:0/620804966 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe648043070 con 0x55cd54e244f0 2024-03-16T11:07:37.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe659012740 10 client.15897 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:37.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.880+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55cd54a57170 con 0x55cd54e244f0 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.881+0000 7fe62f7fe640 1 -- 172.21.15.78:0/620804966 <== mon.0 v2:172.21.15.78:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe6480045b0 con 0x55cd54e244f0 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.881+0000 7fe659012740 10 client.15897 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.881+0000 7fe659012740 10 client.15897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.881+0000 7fe659012740 20 client.15897 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.881+0000 7fe659012740 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e36610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.881+0000 7fe659012740 4 client.15897 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.881+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd54d90610 con 0x55cd54e34220 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.882+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e36610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.882+0000 7fe62f7fe640 10 client.15897 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:37.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.882+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e36610 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68d8b5f033c01b2a server_cookie=58306187bfe62a5f in_seq=0 out_seq=0 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.883+0000 7fe62f7fe640 1 -- 172.21.15.78:0/620804966 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cd54d90610 con 0x55cd54e34220 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.883+0000 7fe62f7fe640 10 client.15897 handle_command_reply: tid=0 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.883+0000 7fe659012740 1 client.15897 shutdown 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.883+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 msgr2=0x55cd54e303f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e303f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe648006010 tx=0x7fe648005df0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 shutdown_connections 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e36610 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe62f7fe640 0 client.15897 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e303f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1f380 0x55cd54e293b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 --2- 172.21.15.78:0/620804966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1e990 0x55cd54e28e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 >> 172.21.15.78:0/620804966 conn(0x55cd54de6c50 msgr2=0x55cd54e194e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:37.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.884+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 shutdown_connections 2024-03-16T11:07:37.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.885+0000 7fe659012740 1 -- 172.21.15.78:0/620804966 wait complete. 2024-03-16T11:07:37.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.885+0000 7fe659012740 20 client.15897 trim_cache size 0 max 16384 2024-03-16T11:07:37.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 Processor -- start 2024-03-16T11:07:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 -- start start 2024-03-16T11:07:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1e990 0x55cd54e1dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e30380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e3bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe6501b18e0 con 0x55cd54e244f0 2024-03-16T11:07:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6501af1b0 con 0x55cd54e1f380 2024-03-16T11:07:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.915+0000 7fe659012740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6501acfe0 con 0x55cd54e1e990 2024-03-16T11:07:37.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e3bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e3bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50484/0 (socket says 172.21.15.78:50484) 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 -- 172.21.15.78:0/3862578833 learned_addr learned my addr 172.21.15.78:0/3862578833 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1e990 0x55cd54e1dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e30380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 -- 172.21.15.78:0/3862578833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 msgr2=0x55cd54e30380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e30380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 -- 172.21.15.78:0/3862578833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 msgr2=0x55cd54e3bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e3bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe6549c4640 1 -- 172.21.15.78:0/3862578833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd54e1b050 con 0x55cd54e1e990 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.916+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e30380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.917+0000 7fe6549c4640 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1e990 0x55cd54e1dc50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe648002230 tx=0x7fe648030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40cca51d73a6da72 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.917+0000 7fe635ffb640 1 -- 172.21.15.78:0/3862578833 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648059040 con 0x55cd54e1e990 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.917+0000 7fe635ffb640 1 -- 172.21.15.78:0/3862578833 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe648032df0 con 0x55cd54e1e990 2024-03-16T11:07:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.917+0000 7fe635ffb640 1 -- 172.21.15.78:0/3862578833 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648039040 con 0x55cd54e1e990 2024-03-16T11:07:37.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 -- 172.21.15.78:0/3862578833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1e990 msgr2=0x55cd54e1dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1e990 0x55cd54e1dc50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe648002230 tx=0x7fe648030000 comp rx=0 tx=0).stop 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 -- 172.21.15.78:0/3862578833 shutdown_connections 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e244f0 0x55cd54e3bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e30380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 --2- 172.21.15.78:0/3862578833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e1e990 0x55cd54e1dc50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 -- 172.21.15.78:0/3862578833 >> 172.21.15.78:0/3862578833 conn(0x55cd54de6c50 msgr2=0x55cd54e1d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 -- 172.21.15.78:0/3862578833 shutdown_connections 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.918+0000 7fe659012740 1 -- 172.21.15.78:0/3862578833 wait complete. 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.919+0000 7fe659012740 1 -- start start 2024-03-16T11:07:37.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.919+0000 7fe659012740 1 Processor -- start 2024-03-16T11:07:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.919+0000 7fe659012740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e1e990 0x55cd54e2e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe659012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e2ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe659012740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e244f0 0x55cd54e2f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e1e990 0x55cd54e2e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e1e990 0x55cd54e2e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50488/0 (socket says 172.21.15.78:50488) 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 -- 172.21.15.78:0/73456690 learned_addr learned my addr 172.21.15.78:0/73456690 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe6501ad670 con 0x55cd54e1e990 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6501aeae0 con 0x55cd54e1f380 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6501add40 con 0x55cd54e244f0 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe659012740 10 client.0 _ll_register_callbacks cb 0x7fe644009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e244f0 0x55cd54e2f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e2ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 -- 172.21.15.78:0/73456690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e244f0 msgr2=0x55cd54e2f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e244f0 0x55cd54e2f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 -- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 msgr2=0x55cd54e2ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e2ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe6549c4640 1 -- 172.21.15.78:0/73456690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd54e1e190 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.920+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e2ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe6549c4640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e1e990 0x55cd54e2e9d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe648033000 tx=0x7fe648009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a51837bd1d8d27e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe62f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe62f7fe640 1 -- 172.21.15.78:0/73456690 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648059070 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe62f7fe640 1 -- 172.21.15.78:0/73456690 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe648048ba0 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe648032df0 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe62f7fe640 1 -- 172.21.15.78:0/73456690 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe648039040 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe659012740 10 client.15903 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55cd54e9b4b0 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.921+0000 7fe62f7fe640 1 -- 172.21.15.78:0/73456690 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe648032df0 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe62f7fe640 1 -- 172.21.15.78:0/73456690 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe648034ae0 con 0x55cd54e1e990 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe659012740 10 client.15903 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe659012740 10 client.15903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe659012740 20 client.15903 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe659012740 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e9ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe659012740 4 client.15903 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T11:07:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cd54e28c60 con 0x55cd54e34220 2024-03-16T11:07:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.922+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e9ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.923+0000 7fe62f7fe640 10 client.15903 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.923+0000 7fe64f7fe640 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e9ddc0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8983a4699d31db98 server_cookie=fed43b5c43bd55c4 in_seq=0 out_seq=0 2024-03-16T11:07:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.924+0000 7fe62f7fe640 1 -- 172.21.15.78:0/73456690 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cd54e28c60 con 0x55cd54e34220 2024-03-16T11:07:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.924+0000 7fe62f7fe640 10 client.15903 handle_command_reply: tid=0 2024-03-16T11:07:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.924+0000 7fe659012740 1 client.15903 shutdown 2024-03-16T11:07:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.924+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e1e990 msgr2=0x55cd54e2e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.924+0000 7fe659012740 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e1e990 0x55cd54e2e9d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe648033000 tx=0x7fe648009970 comp rx=0 tx=0).stop 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 shutdown_connections 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55cd54e34220 0x55cd54e9ddc0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cd54e244f0 0x55cd54e2f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cd54e1f380 0x55cd54e2ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 --2- 172.21.15.78:0/73456690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd54e1e990 0x55cd54e2e9d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 >> 172.21.15.78:0/73456690 conn(0x55cd54de6c50 msgr2=0x55cd54e3af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe62f7fe640 0 client.15903 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 shutdown_connections 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 1 -- 172.21.15.78:0/73456690 wait complete. 2024-03-16T11:07:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.925+0000 7fe659012740 20 client.15903 trim_cache size 0 max 16384 2024-03-16T11:07:37.928 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:07:37.928 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-16T11:07:37.928 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "42930634-5cb5-495d-b479-1f1fe48b0f0f", 2024-03-16T11:07:37.928 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-16T11:07:37.928 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe6380a2900 msgr2=0x7fe6380a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe6380a2900 0x7fe6380a4dc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6400a44b0 tx=0x7fe6400d4530 comp rx=0 tx=0).stop 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500ffac0 msgr2=0x7fe65019de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500ffac0 0x7fe65019de10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe64000e9c0 tx=0x7fe64000ee80 comp rx=0 tx=0).stop 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 shutdown_connections 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe6380a2900 0x7fe6380a4dc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6501034f0 0x7fe6501a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe650102b30 0x7fe65019e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 --2- 172.21.15.78:0/3875436253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6500ffac0 0x7fe65019de10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 >> 172.21.15.78:0/3875436253 conn(0x7fe6500fb3d0 msgr2=0x7fe650103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 shutdown_connections 2024-03-16T11:07:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:37.926+0000 7fe656427640 1 -- 172.21.15.78:0/3875436253 wait complete. 2024-03-16T11:07:37.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:07:37.942 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:07:38.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 -- 172.21.15.78:0/2183053879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 msgr2=0x7fe5e8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2183053879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e8101c30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009ca0 tx=0x7fe5d802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 -- 172.21.15.78:0/2183053879 shutdown_connections 2024-03-16T11:07:38.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2183053879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e81033c0 0x7fe5e8109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2183053879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5e8102a00 0x7fe5e8102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2183053879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e8101c30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 -- 172.21.15.78:0/2183053879 >> 172.21.15.78:0/2183053879 conn(0x7fe5e80fcd80 msgr2=0x7fe5e80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 -- 172.21.15.78:0/2183053879 shutdown_connections 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.098+0000 7fe5ef787640 1 -- 172.21.15.78:0/2183053879 wait complete. 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 Processor -- start 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 -- start start 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e806cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102a00 0x7fe5e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5e81033c0 0x7fe5e80724b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810b180 con 0x7fe5e8102a00 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810b000 con 0x7fe5e81033c0 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ef787640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810b300 con 0x7fe5e8101810 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ed523640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e806cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ed523640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e806cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57474/0 (socket says 172.21.15.78:57474) 2024-03-16T11:07:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ed523640 1 -- 172.21.15.78:0/2361590859 learned_addr learned my addr 172.21.15.78:0/2361590859 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5edd24640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5e81033c0 0x7fe5e80724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.099+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102a00 0x7fe5e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5ed523640 1 -- 172.21.15.78:0/2361590859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5e81033c0 msgr2=0x7fe5e80724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5ed523640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5e81033c0 0x7fe5e80724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5ed523640 1 -- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102a00 msgr2=0x7fe5e806d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5ed523640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102a00 0x7fe5e806d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5ed523640 1 -- 172.21.15.78:0/2361590859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e8072b20 con 0x7fe5e8101810 2024-03-16T11:07:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5ed523640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e806cf10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe5d802c700 tx=0x7fe5d8032810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5e67fc640 1 -- 172.21.15.78:0/2361590859 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8039040 con 0x7fe5e8101810 2024-03-16T11:07:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.100+0000 7fe5e67fc640 1 -- 172.21.15.78:0/2361590859 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5d8044070 con 0x7fe5e8101810 2024-03-16T11:07:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.101+0000 7fe5e67fc640 1 -- 172.21.15.78:0/2361590859 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8004750 con 0x7fe5e8101810 2024-03-16T11:07:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.101+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e8072db0 con 0x7fe5e8101810 2024-03-16T11:07:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.101+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e80732b0 con 0x7fe5e8101810 2024-03-16T11:07:38.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.103+0000 7fe5e67fc640 1 -- 172.21.15.78:0/2361590859 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe5d8004040 con 0x7fe5e8101810 2024-03-16T11:07:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.104+0000 7fe5e67fc640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe5d00a2880 0x7fe5d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.104+0000 7fe5f2372740 1 Processor -- start 2024-03-16T11:07:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.104+0000 7fe5e67fc640 1 -- 172.21.15.78:0/2361590859 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe5d80f13f0 con 0x7fe5e8101810 2024-03-16T11:07:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.104+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe5d00a2880 0x7fe5d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.104+0000 7fe5f2372740 1 -- start start 2024-03-16T11:07:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166ec590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5f2372740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556f16651500 con 0x556f166ecb60 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5f2372740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f165f9be0 con 0x556f166f1cd0 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5f2372740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556f1664d5a0 con 0x556f166ec170 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41156/0 (socket says 172.21.15.78:41156) 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 -- 172.21.15.78:0/1740790417 learned_addr learned my addr 172.21.15.78:0/1740790417 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 -- 172.21.15.78:0/1740790417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 msgr2=0x556f166ec590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5ed523640 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166ec590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166ec590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 -- 172.21.15.78:0/1740790417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 msgr2=0x556f166f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 -- 172.21.15.78:0/1740790417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1660d4b0 con 0x556f166f1cd0 2024-03-16T11:07:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe5d00a2880 0x7fe5d00a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe5e806e420 tx=0x7fe5dc0103d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.105+0000 7fe5edd24640 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f4160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00099e0 tx=0x7fe5e0009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937a03b2112b8a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.107+0000 7fe5cf7fe640 1 -- 172.21.15.78:0/1740790417 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0017070 con 0x556f166f1cd0 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.107+0000 7fe5cf7fe640 1 -- 172.21.15.78:0/1740790417 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5e0013470 con 0x556f166f1cd0 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.107+0000 7fe5cf7fe640 1 -- 172.21.15.78:0/1740790417 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0012610 con 0x556f166f1cd0 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 -- 172.21.15.78:0/1740790417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 msgr2=0x556f166f4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f4160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00099e0 tx=0x7fe5e0009ea0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 -- 172.21.15.78:0/1740790417 shutdown_connections 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f4160 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1740790417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166ec590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 -- 172.21.15.78:0/1740790417 >> 172.21.15.78:0/1740790417 conn(0x556f166e76e0 msgr2=0x556f166e7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 -- 172.21.15.78:0/1740790417 shutdown_connections 2024-03-16T11:07:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.108+0000 7fe5f2372740 1 -- 172.21.15.78:0/1740790417 wait complete. 2024-03-16T11:07:38.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 -- start start 2024-03-16T11:07:38.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 Processor -- start 2024-03-16T11:07:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166f6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5ecd22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5ecd22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50526/0 (socket says 172.21.15.78:50526) 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5ed523640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5ed523640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166f6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41162/0 (socket says 172.21.15.78:41162) 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1384043086 learned_addr learned my addr 172.21.15.78:0/1384043086 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556f166f4c80 con 0x556f166ecb60 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f1660d160 con 0x556f166ec170 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.109+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556f166f4e00 con 0x556f166f1cd0 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5f2372740 10 client.0 _ll_register_callbacks cb 0x556f165f1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5edd24640 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1384043086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 msgr2=0x556f166fdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1384043086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 msgr2=0x556f166f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1384043086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f1642f7d0 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f6c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc010da0 tx=0x7fe5dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7256ea3386fad19 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ccff9640 1 -- 172.21.15.78:0/1384043086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc01f9b0 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5ccff9640 1 -- 172.21.15.78:0/1384043086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe5dc01fb50 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.110+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e0013470 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5ccff9640 1 -- 172.21.15.78:0/1384043086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe5dc01fd50 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 10 client.15912 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556f163653f0 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5ccff9640 1 -- 172.21.15.78:0/1384043086 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5e0013470 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5ccff9640 1 -- 172.21.15.78:0/1384043086 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe5dc027400 con 0x556f166ecb60 2024-03-16T11:07:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 10 client.15912 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:38.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 10 client.15912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:38.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 20 client.15912 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:38.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f167026a0 0x556f16704b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 4 client.15912 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:38.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.111+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f1665ea40 con 0x556f167026a0 2024-03-16T11:07:38.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.112+0000 7fe5ed523640 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f167026a0 0x556f16704b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.113+0000 7fe5ed523640 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f167026a0 0x556f16704b60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46e4282cbe83b1ad server_cookie=8d67062297edca0f in_seq=0 out_seq=0 2024-03-16T11:07:38.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.113+0000 7fe5ccff9640 10 client.15912 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5ccff9640 1 -- 172.21.15.78:0/1384043086 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f1665ea40 con 0x556f167026a0 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5ccff9640 10 client.15912 handle_command_reply: tid=0 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 client.15912 shutdown 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 msgr2=0x556f166f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f6c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc010da0 tx=0x7fe5dc004730 comp rx=0 tx=0).stop 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 shutdown_connections 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f167026a0 0x556f16704b60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166f6c80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1384043086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5ccff9640 0 client.15912 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 >> 172.21.15.78:0/1384043086 conn(0x556f166e76e0 msgr2=0x556f166e6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 shutdown_connections 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.114+0000 7fe5f2372740 1 -- 172.21.15.78:0/1384043086 wait complete. 2024-03-16T11:07:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.115+0000 7fe5f2372740 20 client.15912 trim_cache size 0 max 16384 2024-03-16T11:07:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.143+0000 7fe5f2372740 1 Processor -- start 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.143+0000 7fe5f2372740 1 -- start start 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5f2372740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81ba1f0 con 0x556f166ecb60 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5f2372740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81b7ac0 con 0x556f166f1cd0 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5f2372740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81b58f0 con 0x556f166ec170 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50538/0 (socket says 172.21.15.78:50538) 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1132139926 learned_addr learned my addr 172.21.15.78:0/1132139926 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5edd24640 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ed523640 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1132139926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 msgr2=0x556f166f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1132139926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 msgr2=0x556f166f56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.144+0000 7fe5ecd22640 1 -- 172.21.15.78:0/1132139926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f163250f0 con 0x556f166ecb60 2024-03-16T11:07:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.145+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fdc80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc010e40 tx=0x7fe5dc013570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30db3ce64f8a438b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.145+0000 7fe5ceffd640 1 -- 172.21.15.78:0/1132139926 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc036070 con 0x556f166ecb60 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.145+0000 7fe5ceffd640 1 -- 172.21.15.78:0/1132139926 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5dc027d30 con 0x556f166ecb60 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.145+0000 7fe5ceffd640 1 -- 172.21.15.78:0/1132139926 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc03b750 con 0x556f166ecb60 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 -- 172.21.15.78:0/1132139926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 msgr2=0x556f166fdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fdc80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc010e40 tx=0x7fe5dc013570 comp rx=0 tx=0).stop 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 -- 172.21.15.78:0/1132139926 shutdown_connections 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166f1cd0 0x556f166f56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fdc80 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 --2- 172.21.15.78:0/1132139926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166ec170 0x556f166f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 -- 172.21.15.78:0/1132139926 >> 172.21.15.78:0/1132139926 conn(0x556f166e76e0 msgr2=0x556f166ee130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 -- 172.21.15.78:0/1132139926 shutdown_connections 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.146+0000 7fe5f2372740 1 -- 172.21.15.78:0/1132139926 wait complete. 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 -- start start 2024-03-16T11:07:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 Processor -- start 2024-03-16T11:07:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166facf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81b5f80 con 0x556f166ecb60 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81b73f0 con 0x556f166ec170 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81b6650 con 0x556f166f1cd0 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5f2372740 10 client.0 _ll_register_callbacks cb 0x7fe5dc010c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5ecd22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5ed523640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166facf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5ecd22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50554/0 (socket says 172.21.15.78:50554) 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5edd24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.147+0000 7fe5ecd22640 1 -- 172.21.15.78:0/522767387 learned_addr learned my addr 172.21.15.78:0/522767387 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5edd24640 1 -- 172.21.15.78:0/522767387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 msgr2=0x556f166facf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5edd24640 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166facf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5edd24640 1 -- 172.21.15.78:0/522767387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 msgr2=0x556f166fb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5edd24640 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5edd24640 1 -- 172.21.15.78:0/522767387 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f166189d0 con 0x556f166f1cd0 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5ecd22640 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5edd24640 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fc1f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0013ee0 tx=0x7fe5e0009410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc0542e24ae9eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5ccff9640 1 -- 172.21.15.78:0/522767387 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e002b060 con 0x556f166f1cd0 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5ccff9640 1 -- 172.21.15.78:0/522767387 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe5e0013700 con 0x556f166f1cd0 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.148+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5dc027d30 con 0x556f166f1cd0 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5ccff9640 1 -- 172.21.15.78:0/522767387 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe5e0017040 con 0x556f166f1cd0 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5f2372740 10 client.25729 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556f166fd990 con 0x556f166f1cd0 2024-03-16T11:07:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5ccff9640 1 -- 172.21.15.78:0/522767387 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5dc027d30 con 0x556f166f1cd0 2024-03-16T11:07:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5ccff9640 1 -- 172.21.15.78:0/522767387 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe5e0030070 con 0x556f166f1cd0 2024-03-16T11:07:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5f2372740 10 client.25729 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5f2372740 10 client.25729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.149+0000 7fe5f2372740 20 client.25729 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.150+0000 7fe5f2372740 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f16768aa0 0x556f1676ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.150+0000 7fe5f2372740 4 client.25729 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.150+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556f16658d90 con 0x556f16768aa0 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.150+0000 7fe5ed523640 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f16768aa0 0x556f1676ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.151+0000 7fe5ccff9640 10 client.25729 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.151+0000 7fe5ed523640 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f16768aa0 0x556f1676ae90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c82255a56bc7a8e server_cookie=2477eb0f67f3480a in_seq=0 out_seq=0 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.151+0000 7fe5ccff9640 1 -- 172.21.15.78:0/522767387 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556f16658d90 con 0x556f16768aa0 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.151+0000 7fe5ccff9640 10 client.25729 handle_command_reply: tid=0 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.151+0000 7fe5f2372740 1 client.25729 shutdown 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 msgr2=0x556f166fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fc1f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0013ee0 tx=0x7fe5e0009410 comp rx=0 tx=0).stop 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 shutdown_connections 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f16768aa0 0x556f1676ae90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f166f1cd0 0x556f166fc1f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5ccff9640 0 client.25729 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f166ecb60 0x556f166fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 --2- 172.21.15.78:0/522767387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f166ec170 0x556f166facf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 >> 172.21.15.78:0/522767387 conn(0x556f166e76e0 msgr2=0x556f166f51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.152+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 shutdown_connections 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.153+0000 7fe5f2372740 1 -- 172.21.15.78:0/522767387 wait complete. 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.153+0000 7fe5f2372740 20 client.25729 trim_cache size 0 max 16384 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stdout: "42930634-5cb5-495d-b479-1f1fe48b0f0f": { 2024-03-16T11:07:38.156 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-16T11:07:38.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "42930634-5cb5-495d-b479-1f1fe48b0f0f", 2024-03-16T11:07:38.157 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-16T11:07:38.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:07:38.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:07:38.157 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe5d00a2880 msgr2=0x7fe5d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe5d00a2880 0x7fe5d00a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe5e806e420 tx=0x7fe5dc0103d0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 msgr2=0x7fe5e806cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e806cf10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe5d802c700 tx=0x7fe5d8032810 comp rx=0 tx=0).stop 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 shutdown_connections 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5e81033c0 0x7fe5e80724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102a00 0x7fe5e806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe5d00a2880 0x7fe5d00a4d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 --2- 172.21.15.78:0/2361590859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5e8101810 0x7fe5e806cf10 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe5d802c700 tx=0x7fe5d8032810 comp rx=0 tx=0).stop 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 >> 172.21.15.78:0/2361590859 conn(0x7fe5e80fcd80 msgr2=0x7fe5e810db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 shutdown_connections 2024-03-16T11:07:38.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:07:38.154+0000 7fe5ef787640 1 -- 172.21.15.78:0/2361590859 wait complete. 2024-03-16T11:07:38.172 INFO:tasks.cephfs.filesystem:scrub status for tag:42930634-5cb5-495d-b479-1f1fe48b0f0f - {'path': '/', 'tag': '42930634-5cb5-495d-b479-1f1fe48b0f0f', 'options': 'recursive,force'} 2024-03-16T11:07:38.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:38 smithi078 ceph-mon[32235]: scrub queued for path: / 2024-03-16T11:07:38.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:38 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:38.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:38 smithi078 ceph-mon[32235]: scrub summary: active paths [/] 2024-03-16T11:07:38.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:38 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:38.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:38 smithi098 ceph-mon[35848]: scrub queued for path: / 2024-03-16T11:07:38.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:38 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:38.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:38 smithi098 ceph-mon[35848]: scrub summary: active paths [/] 2024-03-16T11:07:38.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:38 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:38.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:38 smithi153 ceph-mon[35521]: scrub queued for path: / 2024-03-16T11:07:38.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:38 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:38.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:38 smithi153 ceph-mon[35521]: scrub summary: active paths [/] 2024-03-16T11:07:38.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:38 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:07:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:39 smithi098 ceph-mon[35848]: pgmap v398: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 24 MiB/s wr, 2.27k op/s 2024-03-16T11:07:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:39 smithi153 ceph-mon[35521]: pgmap v398: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 24 MiB/s wr, 2.27k op/s 2024-03-16T11:07:39.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:39 smithi078 ceph-mon[32235]: pgmap v398: 97 pgs: 97 active+clean; 258 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 24 MiB/s wr, 2.27k op/s 2024-03-16T11:07:40.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:40 smithi098 ceph-mon[35848]: scrub summary: idle 2024-03-16T11:07:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:40 smithi153 ceph-mon[35521]: scrub summary: idle 2024-03-16T11:07:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:40 smithi078 ceph-mon[32235]: scrub summary: idle 2024-03-16T11:07:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:41 smithi098 ceph-mon[35848]: pgmap v399: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 24 MiB/s wr, 2.92k op/s 2024-03-16T11:07:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:41 smithi153 ceph-mon[35521]: pgmap v399: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 24 MiB/s wr, 2.92k op/s 2024-03-16T11:07:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:41 smithi078 ceph-mon[32235]: pgmap v399: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 24 MiB/s wr, 2.92k op/s 2024-03-16T11:07:43.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:43 smithi098 ceph-mon[35848]: pgmap v400: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 MiB/s wr, 3.76k op/s 2024-03-16T11:07:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:43 smithi153 ceph-mon[35521]: pgmap v400: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 MiB/s wr, 3.76k op/s 2024-03-16T11:07:43.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:43 smithi078 ceph-mon[32235]: pgmap v400: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 MiB/s wr, 3.76k op/s 2024-03-16T11:07:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 -- 172.21.15.78:0/675839115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92900761a0 msgr2=0x7f92900765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 --2- 172.21.15.78:0/675839115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92900761a0 0x7f92900765c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9288009ca0 tx=0x7f928802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 -- 172.21.15.78:0/675839115 shutdown_connections 2024-03-16T11:07:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 --2- 172.21.15.78:0/675839115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 0x7f9290078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 --2- 172.21.15.78:0/675839115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92900773e0 0x7f9290077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 --2- 172.21.15.78:0/675839115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92900761a0 0x7f92900765c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 -- 172.21.15.78:0/675839115 >> 172.21.15.78:0/675839115 conn(0x7f92900fefd0 msgr2=0x7f92901013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 -- 172.21.15.78:0/675839115 shutdown_connections 2024-03-16T11:07:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.286+0000 7f929553b640 1 -- 172.21.15.78:0/675839115 wait complete. 2024-03-16T11:07:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.287+0000 7f929553b640 1 Processor -- start 2024-03-16T11:07:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.287+0000 7f929553b640 1 -- start start 2024-03-16T11:07:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.288+0000 7f929553b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92900761a0 0x7f929019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.288+0000 7f929553b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92900773e0 0x7f929019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.288+0000 7f929553b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 0x7f92901a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.288+0000 7f929553b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9290113970 con 0x7f92900773e0 2024-03-16T11:07:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.288+0000 7f929553b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92901137f0 con 0x7f92900761a0 2024-03-16T11:07:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.288+0000 7f929553b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9290113af0 con 0x7f9290077d80 2024-03-16T11:07:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 0x7f92901a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 0x7f92901a5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:57524/0 (socket says 172.21.15.78:57524) 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 -- 172.21.15.78:0/4106941220 learned_addr learned my addr 172.21.15.78:0/4106941220 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928effd640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92900761a0 0x7f929019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 -- 172.21.15.78:0/4106941220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92900761a0 msgr2=0x7f929019df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92900761a0 0x7f929019df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 -- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92900773e0 msgr2=0x7f929019e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92900773e0 0x7f929019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 -- 172.21.15.78:0/4106941220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92901a5b80 con 0x7f9290077d80 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f928f7fe640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 0x7f92901a5510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f928000e970 tx=0x7f928000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.289+0000 7f9277fff640 1 -- 172.21.15.78:0/4106941220 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000ccc0 con 0x7f9290077d80 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.290+0000 7f9277fff640 1 -- 172.21.15.78:0/4106941220 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f928000ce60 con 0x7f9290077d80 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.290+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92901a5e70 con 0x7f9290077d80 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.290+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92901a63e0 con 0x7f9290077d80 2024-03-16T11:07:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.290+0000 7f9277fff640 1 -- 172.21.15.78:0/4106941220 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280010430 con 0x7f9290077d80 2024-03-16T11:07:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9277fff640 1 -- 172.21.15.78:0/4106941220 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f92800105d0 con 0x7f9290077d80 2024-03-16T11:07:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9298203740 1 Processor -- start 2024-03-16T11:07:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9298203740 1 -- start start 2024-03-16T11:07:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9298203740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789e340 0x56141789e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9298203740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9298203740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614178a3ea0 0x5614178a6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9298203740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561417803a40 con 0x5614178a3ea0 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.293+0000 7f9298203740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5614177ac920 con 0x56141789e340 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f9298203740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5614177ffae0 con 0x56141789ed30 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789e340 0x56141789e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789e340 0x56141789e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:41210/0 (socket says 172.21.15.78:41210) 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 -- 172.21.15.78:0/710024321 learned_addr learned my addr 172.21.15.78:0/710024321 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928effd640 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614178a3ea0 0x5614178a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f9277fff640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f92780a2920 0x7f92780a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 -- 172.21.15.78:0/710024321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 msgr2=0x5614178a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 -- 172.21.15.78:0/710024321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614178a3ea0 msgr2=0x5614178a6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614178a3ea0 0x5614178a6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 -- 172.21.15.78:0/710024321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614177c4460 con 0x56141789e340 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928f7fe640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f92780a2920 0x7f92780a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f9277fff640 1 -- 172.21.15.78:0/4106941220 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f92800cef30 con 0x7f9290077d80 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.294+0000 7f928e7fc640 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789e340 0x56141789e760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f92840103a0 tx=0x7f92840097c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85f1ffda8de31999 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.295+0000 7f92757fa640 1 -- 172.21.15.78:0/710024321 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284018030 con 0x56141789e340 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.295+0000 7f92757fa640 1 -- 172.21.15.78:0/710024321 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92840054d0 con 0x56141789e340 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.295+0000 7f92757fa640 1 -- 172.21.15.78:0/710024321 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284012450 con 0x56141789e340 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.295+0000 7f928f7fe640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f92780a2920 0x7f92780a4de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9280099110 tx=0x7f9280009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 -- 172.21.15.78:0/710024321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789e340 msgr2=0x56141789e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789e340 0x56141789e760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f92840103a0 tx=0x7f92840097c0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 -- 172.21.15.78:0/710024321 shutdown_connections 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614178a3ea0 0x5614178a6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 --2- 172.21.15.78:0/710024321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789e340 0x56141789e760 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 -- 172.21.15.78:0/710024321 >> 172.21.15.78:0/710024321 conn(0x5614178998b0 msgr2=0x561417899cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 -- 172.21.15.78:0/710024321 shutdown_connections 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.296+0000 7f9298203740 1 -- 172.21.15.78:0/710024321 wait complete. 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.297+0000 7f9298203740 1 -- start start 2024-03-16T11:07:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.297+0000 7f9298203740 1 Processor -- start 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.297+0000 7f9298203740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.297+0000 7f9298203740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.297+0000 7f9298203740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f9298203740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5614178a6e50 con 0x56141789e340 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f9298203740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5614177bfd70 con 0x5614178a3ea0 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f9298203740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5614178a6fd0 con 0x56141789ed30 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f9298203740 10 client.0 _ll_register_callbacks cb 0x561417517a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50590/0 (socket says 172.21.15.78:50590) 2024-03-16T11:07:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 -- 172.21.15.78:0/551989609 learned_addr learned my addr 172.21.15.78:0/551989609 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928effd640 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 -- 172.21.15.78:0/551989609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 msgr2=0x5614178a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 -- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 msgr2=0x5614178afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 -- 172.21.15.78:0/551989609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561417518210 con 0x56141789e340 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f928e7fc640 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8680 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9284004850 tx=0x7f92840047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934956849a7b326c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.298+0000 7f9272ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9272ff5640 1 -- 172.21.15.78:0/551989609 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284018040 con 0x56141789e340 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92840054d0 con 0x56141789e340 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9272ff5640 1 -- 172.21.15.78:0/551989609 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9284005000 con 0x56141789e340 2024-03-16T11:07:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9272ff5640 1 -- 172.21.15.78:0/551989609 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9284032050 con 0x56141789e340 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9298203740 10 client.15927 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5614175e4620 con 0x56141789e340 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9272ff5640 1 -- 172.21.15.78:0/551989609 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92840054d0 con 0x56141789e340 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.299+0000 7f9272ff5640 1 -- 172.21.15.78:0/551989609 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f928402a020 con 0x56141789e340 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f9298203740 10 client.15927 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f9298203740 10 client.15927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f9298203740 20 client.15927 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f9298203740 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x5614178b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f9298203740 4 client.15927 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561417810f80 con 0x5614178b42e0 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f928effd640 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x5614178b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f928effd640 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x5614178b66d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a60be41017d1fc server_cookie=1f93458c7d296227 in_seq=0 out_seq=0 2024-03-16T11:07:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.300+0000 7f9272ff5640 10 client.15927 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.301+0000 7f9272ff5640 1 -- 172.21.15.78:0/551989609 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561417810f80 con 0x5614178b42e0 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.301+0000 7f9272ff5640 10 client.15927 handle_command_reply: tid=0 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.301+0000 7f9298203740 1 client.15927 shutdown 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 msgr2=0x5614178a8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8680 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9284004850 tx=0x7f92840047a0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 shutdown_connections 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x5614178b66d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9272ff5640 0 client.15927 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 --2- 172.21.15.78:0/551989609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8680 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.302+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 >> 172.21.15.78:0/551989609 conn(0x5614178998b0 msgr2=0x561417898bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.303+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 shutdown_connections 2024-03-16T11:07:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.303+0000 7f9298203740 1 -- 172.21.15.78:0/551989609 wait complete. 2024-03-16T11:07:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.303+0000 7f9298203740 20 client.15927 trim_cache size 0 max 16384 2024-03-16T11:07:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f9298203740 1 Processor -- start 2024-03-16T11:07:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f9298203740 1 -- start start 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f9298203740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f9298203740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f9298203740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178a87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50600/0 (socket says 172.21.15.78:50600) 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f928e7fc640 1 -- 172.21.15.78:0/303180397 learned_addr learned my addr 172.21.15.78:0/303180397 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f92901b1b40 con 0x56141789e340 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f928effd640 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.329+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92901af410 con 0x5614178a3ea0 2024-03-16T11:07:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92901ad240 con 0x56141789ed30 2024-03-16T11:07:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f928e7fc640 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178a87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f928e7fc640 1 -- 172.21.15.78:0/303180397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 msgr2=0x5614178afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f928e7fc640 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f928e7fc640 1 -- 172.21.15.78:0/303180397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 msgr2=0x5614178a87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f928e7fc640 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178a87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f928e7fc640 1 -- 172.21.15.78:0/303180397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56141789e720 con 0x56141789e340 2024-03-16T11:07:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.330+0000 7f928e7fc640 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8290 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9284004f60 tx=0x7f9284001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af18340bffa3f043 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.331+0000 7f9274ff9640 1 -- 172.21.15.78:0/303180397 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928403b070 con 0x56141789e340 2024-03-16T11:07:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.331+0000 7f9274ff9640 1 -- 172.21.15.78:0/303180397 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9284021030 con 0x56141789e340 2024-03-16T11:07:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.331+0000 7f9274ff9640 1 -- 172.21.15.78:0/303180397 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284005000 con 0x56141789e340 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 msgr2=0x5614178a8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8290 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9284004f60 tx=0x7f9284001e50 comp rx=0 tx=0).stop 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 shutdown_connections 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614178a3ea0 0x5614178a87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56141789ed30 0x5614178afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 --2- 172.21.15.78:0/303180397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178a8290 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 >> 172.21.15.78:0/303180397 conn(0x5614178998b0 msgr2=0x561417894a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 shutdown_connections 2024-03-16T11:07:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.332+0000 7f9298203740 1 -- 172.21.15.78:0/303180397 wait complete. 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 -- start start 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 Processor -- start 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178aca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789ed30 0x5614178acf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5614178a3ea0 0x5614179185c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f92901ad8d0 con 0x56141789e340 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92901aed40 con 0x56141789ed30 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92901adfa0 con 0x5614178a3ea0 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f9298203740 10 client.0 _ll_register_callbacks cb 0x7f9288037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.333+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178aca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50604/0 (socket says 172.21.15.78:50604) 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 -- 172.21.15.78:0/3871478722 learned_addr learned my addr 172.21.15.78:0/3871478722 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928effd640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789ed30 0x5614178acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5614178a3ea0 0x5614179185c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 -- 172.21.15.78:0/3871478722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5614178a3ea0 msgr2=0x5614179185c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5614178a3ea0 0x5614179185c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 -- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789ed30 msgr2=0x5614178acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789ed30 0x5614178acf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 -- 172.21.15.78:0/3871478722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561417918b00 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928effd640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789ed30 0x5614178acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f928e7fc640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178aca00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f92840059f0 tx=0x7f92840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1034f4d24896cea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f9272ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.334+0000 7f9272ff5640 1 -- 172.21.15.78:0/3871478722 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284018030 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9272ff5640 1 -- 172.21.15.78:0/3871478722 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f928401d870 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9284021030 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9272ff5640 1 -- 172.21.15.78:0/3871478722 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f928403b070 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 10 client.15933 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561417919cd0 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9272ff5640 1 -- 172.21.15.78:0/3871478722 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9284021030 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9272ff5640 1 -- 172.21.15.78:0/3871478722 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f928402a040 con 0x56141789e340 2024-03-16T11:07:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 10 client.15933 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 10 client.15933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 20 client.15933 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x56141791c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 4 client.15933 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56141780b2d0 con 0x5614178b42e0 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.335+0000 7f928effd640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x56141791c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.336+0000 7f928effd640 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x56141791c630 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6bd731be4adb4cb server_cookie=663773dc6c530455 in_seq=0 out_seq=0 2024-03-16T11:07:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.336+0000 7f9272ff5640 10 client.15933 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9272ff5640 1 -- 172.21.15.78:0/3871478722 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56141780b2d0 con 0x5614178b42e0 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9272ff5640 10 client.15933 handle_command_reply: tid=0 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 client.15933 shutdown 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 msgr2=0x5614178aca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178aca00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f92840059f0 tx=0x7f92840046c0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 shutdown_connections 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5614178b42e0 0x56141791c630 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5614178a3ea0 0x5614179185c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56141789ed30 0x5614178acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 --2- 172.21.15.78:0/3871478722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56141789e340 0x5614178aca00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 >> 172.21.15.78:0/3871478722 conn(0x5614178998b0 msgr2=0x5614178a8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9272ff5640 0 client.15933 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 shutdown_connections 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 1 -- 172.21.15.78:0/3871478722 wait complete. 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.358+0000 7f9298203740 20 client.15933 trim_cache size 0 max 16384 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f92780a2920 msgr2=0x7f92780a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f92780a2920 0x7f92780a4de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9280099110 tx=0x7f9280009720 comp rx=0 tx=0).stop 2024-03-16T11:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 msgr2=0x7f92901a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 0x7f92901a5510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f928000e970 tx=0x7f928000ed70 comp rx=0 tx=0).stop 2024-03-16T11:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 shutdown_connections 2024-03-16T11:07:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f92780a2920 0x7f92780a4de0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9290077d80 0x7f92901a5510 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92900773e0 0x7f929019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 --2- 172.21.15.78:0/4106941220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92900761a0 0x7f929019df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 >> 172.21.15.78:0/4106941220 conn(0x7f92900fefd0 msgr2=0x7f9290100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 shutdown_connections 2024-03-16T11:07:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:45.359+0000 7f929553b640 1 -- 172.21.15.78:0/4106941220 wait complete. 2024-03-16T11:07:45.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:45 smithi098 ceph-mon[35848]: pgmap v401: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 4.17k op/s 2024-03-16T11:07:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:45 smithi153 ceph-mon[35521]: pgmap v401: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 4.17k op/s 2024-03-16T11:07:45.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:45 smithi078 ceph-mon[32235]: pgmap v401: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 4.17k op/s 2024-03-16T11:07:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:47 smithi098 ceph-mon[35848]: pgmap v402: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.55k op/s 2024-03-16T11:07:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:47 smithi153 ceph-mon[35521]: pgmap v402: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.55k op/s 2024-03-16T11:07:47.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:47 smithi078 ceph-mon[32235]: pgmap v402: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.55k op/s 2024-03-16T11:07:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:49 smithi098 ceph-mon[35848]: pgmap v403: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 18 MiB/s wr, 4.19k op/s 2024-03-16T11:07:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:49 smithi153 ceph-mon[35521]: pgmap v403: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 18 MiB/s wr, 4.19k op/s 2024-03-16T11:07:49.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:49 smithi078 ceph-mon[32235]: pgmap v403: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 18 MiB/s wr, 4.19k op/s 2024-03-16T11:07:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:51 smithi098 ceph-mon[35848]: pgmap v404: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 19 MiB/s wr, 4.28k op/s 2024-03-16T11:07:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:51 smithi153 ceph-mon[35521]: pgmap v404: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 19 MiB/s wr, 4.28k op/s 2024-03-16T11:07:51.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:51 smithi078 ceph-mon[32235]: pgmap v404: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 19 MiB/s wr, 4.28k op/s 2024-03-16T11:07:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:53 smithi098 ceph-mon[35848]: pgmap v405: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.51k op/s 2024-03-16T11:07:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:53 smithi153 ceph-mon[35521]: pgmap v405: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.51k op/s 2024-03-16T11:07:53.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:53 smithi078 ceph-mon[32235]: pgmap v405: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.51k op/s 2024-03-16T11:07:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:55 smithi098 ceph-mon[35848]: pgmap v406: 97 pgs: 97 active+clean; 258 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 19 MiB/s wr, 4.33k op/s 2024-03-16T11:07:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:55 smithi153 ceph-mon[35521]: pgmap v406: 97 pgs: 97 active+clean; 258 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 19 MiB/s wr, 4.33k op/s 2024-03-16T11:07:55.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:55 smithi078 ceph-mon[32235]: pgmap v406: 97 pgs: 97 active+clean; 258 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 19 MiB/s wr, 4.33k op/s 2024-03-16T11:07:57.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:57 smithi098 ceph-mon[35848]: pgmap v407: 97 pgs: 97 active+clean; 258 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.60k op/s 2024-03-16T11:07:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:57 smithi153 ceph-mon[35521]: pgmap v407: 97 pgs: 97 active+clean; 258 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.60k op/s 2024-03-16T11:07:57.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:57 smithi078 ceph-mon[32235]: pgmap v407: 97 pgs: 97 active+clean; 258 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.60k op/s 2024-03-16T11:07:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 -- 172.21.15.78:0/2018904672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c101d00 msgr2=0x7f635c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 --2- 172.21.15.78:0/2018904672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c101d00 0x7f635c102120 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6358009930 tx=0x7f635802bc50 comp rx=0 tx=0).stop 2024-03-16T11:07:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 -- 172.21.15.78:0/2018904672 shutdown_connections 2024-03-16T11:07:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 --2- 172.21.15.78:0/2018904672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c101360 0x7f635c108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 --2- 172.21.15.78:0/2018904672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c1009a0 0x7f635c100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 --2- 172.21.15.78:0/2018904672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c101d00 0x7f635c102120 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 -- 172.21.15.78:0/2018904672 >> 172.21.15.78:0/2018904672 conn(0x7f635c0fb430 msgr2=0x7f635c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.489+0000 7f6363e0b640 1 -- 172.21.15.78:0/2018904672 shutdown_connections 2024-03-16T11:07:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.490+0000 7f6363e0b640 1 -- 172.21.15.78:0/2018904672 wait complete. 2024-03-16T11:07:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.490+0000 7f6363e0b640 1 Processor -- start 2024-03-16T11:07:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.490+0000 7f6363e0b640 1 -- start start 2024-03-16T11:07:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.490+0000 7f6363e0b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c1009a0 0x7f635c199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.490+0000 7f6363e0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c101360 0x7f635c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.490+0000 7f6363e0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c101d00 0x7f635c1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f6363e0b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f635c10c310 con 0x7f635c1009a0 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f6363e0b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f635c10c190 con 0x7f635c101360 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f6363e0b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f635c10c490 con 0x7f635c101d00 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c101360 0x7f635c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f6361ba7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c1009a0 0x7f635c199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c101360 0x7f635c19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:37068/0 (socket says 172.21.15.78:37068) 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 -- 172.21.15.78:0/3460808750 learned_addr learned my addr 172.21.15.78:0/3460808750 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63623a8640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c101d00 0x7f635c1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 -- 172.21.15.78:0/3460808750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c101d00 msgr2=0x7f635c1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c101d00 0x7f635c1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 -- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c1009a0 msgr2=0x7f635c199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c1009a0 0x7f635c199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 -- 172.21.15.78:0/3460808750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635c1a1930 con 0x7f635c101360 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.491+0000 7f63613a6640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c101360 0x7f635c19a260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f634c00c890 tx=0x7f634c00cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.492+0000 7f6352ffd640 1 -- 172.21.15.78:0/3460808750 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c004220 con 0x7f635c101360 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.492+0000 7f6352ffd640 1 -- 172.21.15.78:0/3460808750 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f634c0043c0 con 0x7f635c101360 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.492+0000 7f6352ffd640 1 -- 172.21.15.78:0/3460808750 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c004a00 con 0x7f635c101360 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.492+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f635c1a1c20 con 0x7f635c101360 2024-03-16T11:07:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.492+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f635c1a20f0 con 0x7f635c101360 2024-03-16T11:07:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.495+0000 7f6352ffd640 1 -- 172.21.15.78:0/3460808750 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f634c004ba0 con 0x7f635c101360 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 Processor -- start 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 -- start start 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639d7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639da340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f363926a40 con 0x55f3639d2350 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3638cf920 con 0x55f3639d2d40 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63669f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f363922ae0 con 0x55f3639d7eb0 2024-03-16T11:07:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f6352ffd640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63480a2920 0x7f63480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63623a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f6361ba7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639d7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f6361ba7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639d7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:37084/0 (socket says 172.21.15.78:37084) 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f63623a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639d2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60298/0 (socket says 172.21.15.78:60298) 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f6361ba7640 1 -- 172.21.15.78:0/2306878129 learned_addr learned my addr 172.21.15.78:0/2306878129 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.496+0000 7f6352ffd640 1 -- 172.21.15.78:0/3460808750 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f634c0ce690 con 0x7f635c101360 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6361ba7640 1 -- 172.21.15.78:0/2306878129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 msgr2=0x55f3639da340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6361ba7640 1 --2- 172.21.15.78:0/2306878129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639da340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6361ba7640 1 -- 172.21.15.78:0/2306878129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 msgr2=0x55f3639d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6361ba7640 1 --2- 172.21.15.78:0/2306878129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6361ba7640 1 -- 172.21.15.78:0/2306878129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3638e7460 con 0x55f3639d2d40 2024-03-16T11:07:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f63623a8640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63480a2920 0x7f63480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6361ba7640 1 --2- 172.21.15.78:0/2306878129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639d7840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f635802c860 tx=0x7f635802c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=967a3d5c8cbe7822 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f63623a8640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63480a2920 0x7f63480a4de0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55f3639d35a0 tx=0x7f6354004a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6347fff640 1 -- 172.21.15.78:0/2306878129 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358045020 con 0x55f3639d2d40 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6347fff640 1 -- 172.21.15.78:0/2306878129 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f635802cc30 con 0x55f3639d2d40 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.497+0000 7f6347fff640 1 -- 172.21.15.78:0/2306878129 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358004920 con 0x55f3639d2d40 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 -- 172.21.15.78:0/2306878129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 msgr2=0x55f3639d7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 --2- 172.21.15.78:0/2306878129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639d7840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f635802c860 tx=0x7f635802c980 comp rx=0 tx=0).stop 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 -- 172.21.15.78:0/2306878129 shutdown_connections 2024-03-16T11:07:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 --2- 172.21.15.78:0/2306878129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639da340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 --2- 172.21.15.78:0/2306878129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639d7840 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 --2- 172.21.15.78:0/2306878129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 -- 172.21.15.78:0/2306878129 >> 172.21.15.78:0/2306878129 conn(0x55f3639bc8b0 msgr2=0x55f3639bccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.498+0000 7f63669f3740 1 -- 172.21.15.78:0/2306878129 shutdown_connections 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.499+0000 7f63669f3740 1 -- 172.21.15.78:0/2306878129 wait complete. 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.499+0000 7f63669f3740 1 -- start start 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.499+0000 7f63669f3740 1 Processor -- start 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63669f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639dc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63669f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2d40 0x55f3639dcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63669f3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d7eb0 0x55f3639e3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63669f3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f3639dae00 con 0x55f3639d2350 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63669f3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3638e2d70 con 0x55f3639d7eb0 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63669f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f3639daf80 con 0x55f3639d2d40 2024-03-16T11:07:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f6361ba7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639dc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f6361ba7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639dc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60302/0 (socket says 172.21.15.78:60302) 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f6361ba7640 1 -- 172.21.15.78:0/1274833464 learned_addr learned my addr 172.21.15.78:0/1274833464 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63623a8640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d7eb0 0x55f3639e3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63669f3740 10 client.0 _ll_register_callbacks cb 0x55f36363aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63613a6640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2d40 0x55f3639dcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63623a8640 1 -- 172.21.15.78:0/1274833464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2d40 msgr2=0x55f3639dcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63623a8640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2d40 0x55f3639dcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63623a8640 1 -- 172.21.15.78:0/1274833464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 msgr2=0x55f3639dc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63623a8640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639dc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f63623a8640 1 -- 172.21.15.78:0/1274833464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f36363b210 con 0x55f3639d7eb0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.500+0000 7f6361ba7640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639dc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63623a8640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d7eb0 0x55f3639e3c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6354007de0 tx=0x7f6354004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d68fec47ab869164 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63457fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63457fa640 1 -- 172.21.15.78:0/1274833464 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354028060 con 0x55f3639d7eb0 2024-03-16T11:07:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63457fa640 1 -- 172.21.15.78:0/1274833464 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f635401a0d0 con 0x55f3639d7eb0 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f635802cc30 con 0x55f3639d7eb0 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63457fa640 1 -- 172.21.15.78:0/1274833464 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6354014040 con 0x55f3639d7eb0 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63669f3740 10 client.25775 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f363707620 con 0x55f3639d7eb0 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63457fa640 1 -- 172.21.15.78:0/1274833464 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f635802cc30 con 0x55f3639d7eb0 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63457fa640 1 -- 172.21.15.78:0/1274833464 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f635402d070 con 0x55f3639d7eb0 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63669f3740 10 client.25775 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63669f3740 10 client.25775 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.501+0000 7f63669f3740 20 client.25775 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.502+0000 7f63669f3740 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f3639ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.502+0000 7f63669f3740 4 client.25775 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.502+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f363933f80 con 0x55f3639e84e0 2024-03-16T11:07:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.502+0000 7f6361ba7640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f3639ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.502+0000 7f63457fa640 10 client.25775 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.502+0000 7f6361ba7640 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f3639ea910 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65a1f5dbc1a21471 server_cookie=a8805038de271582 in_seq=0 out_seq=0 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.503+0000 7f63457fa640 1 -- 172.21.15.78:0/1274833464 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f363933f80 con 0x55f3639e84e0 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.503+0000 7f63457fa640 10 client.25775 handle_command_reply: tid=0 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.503+0000 7f63669f3740 1 client.25775 shutdown 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d7eb0 msgr2=0x55f3639e3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d7eb0 0x55f3639e3c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6354007de0 tx=0x7f6354004ad0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 shutdown_connections 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f3639ea910 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d7eb0 0x55f3639e3c60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63457fa640 0 client.25775 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2d40 0x55f3639dcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 --2- 172.21.15.78:0/1274833464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639dc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 >> 172.21.15.78:0/1274833464 conn(0x55f3639bc8b0 msgr2=0x55f3639bbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 shutdown_connections 2024-03-16T11:07:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.504+0000 7f63669f3740 1 -- 172.21.15.78:0/1274833464 wait complete. 2024-03-16T11:07:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.505+0000 7f63669f3740 20 client.25775 trim_cache size 0 max 16384 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 Processor -- start 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 -- start start 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2350 0x55f3639c0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d7eb0 0x55f3639ef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f635c1a4fa0 con 0x55f3639d7eb0 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f635c1a2870 con 0x55f3639d2d40 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.526+0000 7f63669f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f635c071d90 con 0x55f3639d2350 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f6361ba7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2350 0x55f3639c0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f6361ba7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2350 0x55f3639c0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:39978/0 (socket says 172.21.15.78:39978) 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f6361ba7640 1 -- 172.21.15.78:0/1716955372 learned_addr learned my addr 172.21.15.78:0/1716955372 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63613a6640 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63623a8640 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d7eb0 0x55f3639ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63613a6640 1 -- 172.21.15.78:0/1716955372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2350 msgr2=0x55f3639c0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63613a6640 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2350 0x55f3639c0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63613a6640 1 -- 172.21.15.78:0/1716955372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d7eb0 msgr2=0x55f3639ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63613a6640 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d7eb0 0x55f3639ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63613a6640 1 -- 172.21.15.78:0/1716955372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3639d3120 con 0x55f3639d2d40 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63613a6640 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e3c20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f634c09b070 tx=0x7f634c09ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4c539b46d8d766a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63477fe640 1 -- 172.21.15.78:0/1716955372 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c01d040 con 0x55f3639d2d40 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63477fe640 1 -- 172.21.15.78:0/1716955372 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f634c09f180 con 0x55f3639d2d40 2024-03-16T11:07:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.527+0000 7f63477fe640 1 -- 172.21.15.78:0/1716955372 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c095bc0 con 0x55f3639d2d40 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.528+0000 7f63669f3740 1 -- 172.21.15.78:0/1716955372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 msgr2=0x55f3639e3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.528+0000 7f63669f3740 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e3c20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f634c09b070 tx=0x7f634c09ab60 comp rx=0 tx=0).stop 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.528+0000 7f63669f3740 1 -- 172.21.15.78:0/1716955372 shutdown_connections 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.528+0000 7f63669f3740 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d7eb0 0x55f3639ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.528+0000 7f63669f3740 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e3c20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.528+0000 7f63669f3740 1 --2- 172.21.15.78:0/1716955372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d2350 0x55f3639c0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.528+0000 7f63669f3740 1 -- 172.21.15.78:0/1716955372 >> 172.21.15.78:0/1716955372 conn(0x55f3639bc8b0 msgr2=0x55f3639bfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.529+0000 7f63669f3740 1 -- 172.21.15.78:0/1716955372 shutdown_connections 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.529+0000 7f63669f3740 1 -- 172.21.15.78:0/1716955372 wait complete. 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.529+0000 7f63669f3740 1 -- start start 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.529+0000 7f63669f3740 1 Processor -- start 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.529+0000 7f63669f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639e14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63669f3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63669f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639e29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63613a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63613a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:37122/0 (socket says 172.21.15.78:37122) 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63613a6640 1 -- 172.21.15.78:0/1166819118 learned_addr learned my addr 172.21.15.78:0/1166819118 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f635c072420 con 0x55f3639d2350 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f635c073890 con 0x55f3639d2d40 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f635c072af0 con 0x55f3639d7eb0 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63623a8640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639e29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f6361ba7640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639e14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63669f3740 10 client.0 _ll_register_callbacks cb 0x7f6354007570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:07:58.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.530+0000 7f63613a6640 1 -- 172.21.15.78:0/1166819118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 msgr2=0x55f3639e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63613a6640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639e29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63613a6640 1 -- 172.21.15.78:0/1166819118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 msgr2=0x55f3639e14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63613a6640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639e14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63613a6640 1 -- 172.21.15.78:0/1166819118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3639e2f90 con 0x55f3639d2d40 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63623a8640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639e29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f6361ba7640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639e14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63613a6640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e19e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f634c097a20 tx=0x7f634c005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c9977df6cc4df server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63457fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.531+0000 7f63457fa640 1 -- 172.21.15.78:0/1166819118 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c07e060 con 0x55f3639d2d40 2024-03-16T11:07:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.532+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f634c09f180 con 0x55f3639d2d40 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.532+0000 7f63457fa640 1 -- 172.21.15.78:0/1166819118 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f634c0b9b30 con 0x55f3639d2d40 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.532+0000 7f63457fa640 1 -- 172.21.15.78:0/1166819118 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f634c01d040 con 0x55f3639d2d40 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.532+0000 7f63669f3740 10 client.25781 fetch_fsmap learned FSMap version 22 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.532+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f363a4e950 con 0x55f3639d2d40 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63457fa640 1 -- 172.21.15.78:0/1166819118 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f634c09f180 con 0x55f3639d2d40 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63457fa640 1 -- 172.21.15.78:0/1166819118 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f634c096f60 con 0x55f3639d2d40 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63669f3740 10 client.25781 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63669f3740 10 client.25781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63669f3740 20 client.25781 populate_metadata read hostname 'smithi078' 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63669f3740 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f363a512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63669f3740 4 client.25781 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f36392e2d0 con 0x55f3639e84e0 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.533+0000 7f6361ba7640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f363a512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.534+0000 7f63457fa640 10 client.25781 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.534+0000 7f6361ba7640 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f363a512c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=504b0348985e656a server_cookie=c59a06d9f00ae432 in_seq=0 out_seq=0 2024-03-16T11:07:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.546+0000 7f63457fa640 1 -- 172.21.15.78:0/1166819118 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f36392e2d0 con 0x55f3639e84e0 2024-03-16T11:07:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.546+0000 7f63457fa640 10 client.25781 handle_command_reply: tid=0 2024-03-16T11:07:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.546+0000 7f63669f3740 1 client.25781 shutdown 2024-03-16T11:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.547+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 msgr2=0x55f3639e19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.547+0000 7f63669f3740 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e19e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f634c097a20 tx=0x7f634c005760 comp rx=0 tx=0).stop 2024-03-16T11:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 shutdown_connections 2024-03-16T11:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55f3639e84e0 0x55f363a512c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3639d7eb0 0x55f3639e29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f3639d2d40 0x55f3639e19e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 --2- 172.21.15.78:0/1166819118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f3639d2350 0x55f3639e14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63457fa640 0 client.25781 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 >> 172.21.15.78:0/1166819118 conn(0x55f3639bc8b0 msgr2=0x55f3639c03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 shutdown_connections 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 1 -- 172.21.15.78:0/1166819118 wait complete. 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.548+0000 7f63669f3740 20 client.25781 trim_cache size 0 max 16384 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:07:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:07:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63480a2920 msgr2=0x7f63480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63480a2920 0x7f63480a4de0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55f3639d35a0 tx=0x7f6354004a60 comp rx=0 tx=0).stop 2024-03-16T11:07:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c101360 msgr2=0x7f635c19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:07:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c101360 0x7f635c19a260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f634c00c890 tx=0x7f634c00cd50 comp rx=0 tx=0).stop 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 shutdown_connections 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f63480a2920 0x7f63480a4de0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c101d00 0x7f635c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f635c101360 0x7f635c19a260 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 --2- 172.21.15.78:0/3460808750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f635c1009a0 0x7f635c199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 >> 172.21.15.78:0/3460808750 conn(0x7f635c0fb430 msgr2=0x7f635c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 shutdown_connections 2024-03-16T11:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:07:58.549+0000 7f6363e0b640 1 -- 172.21.15.78:0/3460808750 wait complete. 2024-03-16T11:07:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:07:59 smithi098 ceph-mon[35848]: pgmap v408: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 18 MiB/s wr, 4.26k op/s 2024-03-16T11:07:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:07:59 smithi153 ceph-mon[35521]: pgmap v408: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 18 MiB/s wr, 4.26k op/s 2024-03-16T11:07:59.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:07:59 smithi078 ceph-mon[32235]: pgmap v408: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 18 MiB/s wr, 4.26k op/s 2024-03-16T11:08:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:01 smithi098 ceph-mon[35848]: pgmap v409: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 19 MiB/s wr, 4.36k op/s 2024-03-16T11:08:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:01 smithi153 ceph-mon[35521]: pgmap v409: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 19 MiB/s wr, 4.36k op/s 2024-03-16T11:08:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:01 smithi078 ceph-mon[32235]: pgmap v409: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 19 MiB/s wr, 4.36k op/s 2024-03-16T11:08:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:03 smithi098 ceph-mon[35848]: pgmap v410: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.53k op/s 2024-03-16T11:08:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:03 smithi153 ceph-mon[35521]: pgmap v410: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.53k op/s 2024-03-16T11:08:03.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:03 smithi078 ceph-mon[32235]: pgmap v410: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.53k op/s 2024-03-16T11:08:05.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:05 smithi098 ceph-mon[35848]: pgmap v411: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 4.31k op/s 2024-03-16T11:08:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:05 smithi153 ceph-mon[35521]: pgmap v411: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 4.31k op/s 2024-03-16T11:08:05.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:05 smithi078 ceph-mon[32235]: pgmap v411: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 4.31k op/s 2024-03-16T11:08:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:07 smithi098 ceph-mon[35848]: pgmap v412: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.55k op/s 2024-03-16T11:08:07.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:07 smithi153 ceph-mon[35521]: pgmap v412: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.55k op/s 2024-03-16T11:08:07.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:07 smithi078 ceph-mon[32235]: pgmap v412: 97 pgs: 97 active+clean; 258 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 4.55k op/s 2024-03-16T11:08:08.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:08:08.173 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:08:08.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 -- 172.21.15.78:0/339995840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c41022d0 msgr2=0x7fe9c4108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 --2- 172.21.15.78:0/339995840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c41022d0 0x7fe9c4108b60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00bd70 tx=0x7fe9bc02dc70 comp rx=0 tx=0).stop 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 -- 172.21.15.78:0/339995840 shutdown_connections 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 --2- 172.21.15.78:0/339995840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c41022d0 0x7fe9c4108b60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 --2- 172.21.15.78:0/339995840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c4101910 0x7fe9c4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 --2- 172.21.15.78:0/339995840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 0x7fe9c4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 -- 172.21.15.78:0/339995840 >> 172.21.15.78:0/339995840 conn(0x7fe9c40fbc90 msgr2=0x7fe9c40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 -- 172.21.15.78:0/339995840 shutdown_connections 2024-03-16T11:08:08.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.329+0000 7fe9cb461640 1 -- 172.21.15.78:0/339995840 wait complete. 2024-03-16T11:08:08.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 Processor -- start 2024-03-16T11:08:08.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 -- start start 2024-03-16T11:08:08.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 0x7fe9c4195600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c4101910 0x7fe9c4195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c41022d0 0x7fe9c419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c410b150 con 0x7fe9c4101910 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c410afd0 con 0x7fe9c41022d0 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9cb461640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c410b2d0 con 0x7fe9c4100720 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 0x7fe9c4195600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9c99fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c41022d0 0x7fe9c419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 0x7fe9c4195600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50406/0 (socket says 172.21.15.78:50406) 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9c91fd640 1 -- 172.21.15.78:0/3638994877 learned_addr learned my addr 172.21.15.78:0/3638994877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.330+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c4101910 0x7fe9c4195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9c91fd640 1 -- 172.21.15.78:0/3638994877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c41022d0 msgr2=0x7fe9c419cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c41022d0 0x7fe9c419cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9c91fd640 1 -- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c4101910 msgr2=0x7fe9c4195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c4101910 0x7fe9c4195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9c91fd640 1 -- 172.21.15.78:0/3638994877 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c419d220 con 0x7fe9c4100720 2024-03-16T11:08:08.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c41022d0 0x7fe9c419cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:08.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 0x7fe9c4195600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400e9c0 tx=0x7fe9b400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9ba7fc640 1 -- 172.21.15.78:0/3638994877 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400ccc0 con 0x7fe9c4100720 2024-03-16T11:08:08.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.331+0000 7fe9ba7fc640 1 -- 172.21.15.78:0/3638994877 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9b40035f0 con 0x7fe9c4100720 2024-03-16T11:08:08.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.332+0000 7fe9ba7fc640 1 -- 172.21.15.78:0/3638994877 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400ce60 con 0x7fe9c4100720 2024-03-16T11:08:08.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.332+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c419d510 con 0x7fe9c4100720 2024-03-16T11:08:08.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.332+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c419d9b0 con 0x7fe9c4100720 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.335+0000 7fe9ce052740 1 Processor -- start 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.335+0000 7fe9ba7fc640 1 -- 172.21.15.78:0/3638994877 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe9b40037a0 con 0x7fe9c4100720 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.335+0000 7fe9ce052740 1 -- start start 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd93590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd98660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fd9b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9ce052740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556f7fcf8500 con 0x556f7fd98cd0 2024-03-16T11:08:08.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9ce052740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f7fca0be0 con 0x556f7fd93170 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9ce052740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556f7fcf45a0 con 0x556f7fd93b60 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fd9b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c99fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd93590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd98660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fd9b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41606/0 (socket says 172.21.15.78:41606) 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 -- 172.21.15.78:0/17735759 learned_addr learned my addr 172.21.15.78:0/17735759 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 -- 172.21.15.78:0/17735759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 msgr2=0x556f7fd98660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/17735759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd98660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 -- 172.21.15.78:0/17735759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 msgr2=0x556f7fd93590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/17735759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd93590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9c89fc640 1 -- 172.21.15.78:0/17735759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f7fcb44b0 con 0x556f7fd98cd0 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.336+0000 7fe9ba7fc640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe9b00a28d0 0x7fe9b00a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.337+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/17735759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fd9b160 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009ca0 tx=0x7fe9c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af13b1d7afb56f91 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.337+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe9b00a28d0 0x7fe9b00a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.337+0000 7fe9ba7fc640 1 -- 172.21.15.78:0/3638994877 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe9b40cee10 con 0x7fe9c4100720 2024-03-16T11:08:08.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.337+0000 7fe9af7fe640 1 -- 172.21.15.78:0/17735759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c001e070 con 0x556f7fd98cd0 2024-03-16T11:08:08.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.337+0000 7fe9af7fe640 1 -- 172.21.15.78:0/17735759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9c0009e50 con 0x556f7fd98cd0 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.337+0000 7fe9af7fe640 1 -- 172.21.15.78:0/17735759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0019690 con 0x556f7fd98cd0 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe9b00a28d0 0x7fe9b00a4d90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x556f7fd943c0 tx=0x7fe9bc00ba50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 -- 172.21.15.78:0/17735759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 msgr2=0x556f7fd9b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 --2- 172.21.15.78:0/17735759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fd9b160 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009ca0 tx=0x7fe9c0011810 comp rx=0 tx=0).stop 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 -- 172.21.15.78:0/17735759 shutdown_connections 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 --2- 172.21.15.78:0/17735759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fd9b160 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 --2- 172.21.15.78:0/17735759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd98660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 --2- 172.21.15.78:0/17735759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd93590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 -- 172.21.15.78:0/17735759 >> 172.21.15.78:0/17735759 conn(0x556f7fd8e6e0 msgr2=0x556f7fd8eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 -- 172.21.15.78:0/17735759 shutdown_connections 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 -- 172.21.15.78:0/17735759 wait complete. 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.338+0000 7fe9ce052740 1 -- start start 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 1 Processor -- start 2024-03-16T11:08:08.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd9d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd9dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556f7fd9bcd0 con 0x556f7fd98cd0 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556f7fcb4160 con 0x556f7fd93170 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556f7fd9be50 con 0x556f7fd93b60 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9ce052740 10 client.0 _ll_register_callbacks cb 0x556f7fc98150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd9d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd9d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36566/0 (socket says 172.21.15.78:36566) 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9c91fd640 1 -- 172.21.15.78:0/199974895 learned_addr learned my addr 172.21.15.78:0/199974895 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.339+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd9dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9c91fd640 1 -- 172.21.15.78:0/199974895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 msgr2=0x556f7fd9dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd9dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9c91fd640 1 -- 172.21.15.78:0/199974895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 msgr2=0x556f7fda4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9c91fd640 1 -- 172.21.15.78:0/199974895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f7fad67d0 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd9d6b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40a3a20 tx=0x7fe9b4097450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73edcf75b3e91945 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9acff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9acff9640 1 -- 172.21.15.78:0/199974895 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b409b2f0 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9c0009e50 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.340+0000 7fe9acff9640 1 -- 172.21.15.78:0/199974895 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe9b40a3ac0 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9acff9640 1 -- 172.21.15.78:0/199974895 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe9b409b110 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 10 client.25793 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556f7fa0c3f0 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9acff9640 1 -- 172.21.15.78:0/199974895 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9c0009e50 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9acff9640 1 -- 172.21.15.78:0/199974895 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe9b4097500 con 0x556f7fd93170 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 10 client.25793 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 20 client.25793 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:08.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fda95a0 0x556f7fdaba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 4 client.25793 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:08.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f7fd05a40 con 0x556f7fda95a0 2024-03-16T11:08:08.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.341+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fda95a0 0x556f7fdaba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.342+0000 7fe9acff9640 10 client.25793 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:08.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.342+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fda95a0 0x556f7fdaba30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0528d88964d7b31 server_cookie=a9167a2161012cd1 in_seq=0 out_seq=0 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.343+0000 7fe9acff9640 1 -- 172.21.15.78:0/199974895 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f7fd05a40 con 0x556f7fda95a0 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.343+0000 7fe9acff9640 10 client.25793 handle_command_reply: tid=0 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.343+0000 7fe9ce052740 1 client.25793 shutdown 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.343+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 msgr2=0x556f7fd9d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.343+0000 7fe9ce052740 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd9d6b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40a3a20 tx=0x7fe9b4097450 comp rx=0 tx=0).stop 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.343+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 shutdown_connections 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fda95a0 0x556f7fdaba30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93b60 0x556f7fd9dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 1 --2- 172.21.15.78:0/199974895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93170 0x556f7fd9d6b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 >> 172.21.15.78:0/199974895 conn(0x556f7fd8e6e0 msgr2=0x556f7fd8d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9acff9640 0 client.25793 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:08.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 shutdown_connections 2024-03-16T11:08:08.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 1 -- 172.21.15.78:0/199974895 wait complete. 2024-03-16T11:08:08.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.344+0000 7fe9ce052740 20 client.25793 trim_cache size 0 max 16384 2024-03-16T11:08:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.374+0000 7fe9ce052740 1 Processor -- start 2024-03-16T11:08:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.374+0000 7fe9ce052740 1 -- start start 2024-03-16T11:08:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd93170 0x556f7fd898e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd98cd0 0x556f7fd9d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9ce052740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41a9060 con 0x556f7fd93170 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9ce052740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41a6930 con 0x556f7fd93b60 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9ce052740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41a4760 con 0x556f7fd98cd0 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd93170 0x556f7fd898e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd93170 0x556f7fd898e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41630/0 (socket says 172.21.15.78:41630) 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 -- 172.21.15.78:0/1937453925 learned_addr learned my addr 172.21.15.78:0/1937453925 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd98cd0 0x556f7fd9d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 -- 172.21.15.78:0/1937453925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd98cd0 msgr2=0x556f7fd9d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd98cd0 0x556f7fd9d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 -- 172.21.15.78:0/1937453925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 msgr2=0x556f7fda4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c91fd640 1 -- 172.21.15.78:0/1937453925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f7f9cc0f0 con 0x556f7fd93170 2024-03-16T11:08:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.375+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd98cd0 0x556f7fd9d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:08:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.376+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd93170 0x556f7fd898e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c6a0 tx=0x7fe9b4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78abedcedc656edc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.376+0000 7fe9aeffd640 1 -- 172.21.15.78:0/1937453925 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4003b30 con 0x556f7fd93170 2024-03-16T11:08:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.376+0000 7fe9aeffd640 1 -- 172.21.15.78:0/1937453925 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9b409beb0 con 0x556f7fd93170 2024-03-16T11:08:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.376+0000 7fe9aeffd640 1 -- 172.21.15.78:0/1937453925 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b40a3ac0 con 0x556f7fd93170 2024-03-16T11:08:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.377+0000 7fe9ce052740 1 -- 172.21.15.78:0/1937453925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd93170 msgr2=0x556f7fd898e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.377+0000 7fe9ce052740 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd93170 0x556f7fd898e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c6a0 tx=0x7fe9b4000bc0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.378+0000 7fe9ce052740 1 -- 172.21.15.78:0/1937453925 shutdown_connections 2024-03-16T11:08:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.378+0000 7fe9ce052740 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd98cd0 0x556f7fd9d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.378+0000 7fe9ce052740 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.378+0000 7fe9ce052740 1 --2- 172.21.15.78:0/1937453925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd93170 0x556f7fd898e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.378+0000 7fe9ce052740 1 -- 172.21.15.78:0/1937453925 >> 172.21.15.78:0/1937453925 conn(0x556f7fd8e6e0 msgr2=0x556f7fd94560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.378+0000 7fe9ce052740 1 -- 172.21.15.78:0/1937453925 shutdown_connections 2024-03-16T11:08:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.378+0000 7fe9ce052740 1 -- 172.21.15.78:0/1937453925 wait complete. 2024-03-16T11:08:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.379+0000 7fe9ce052740 1 -- start start 2024-03-16T11:08:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.379+0000 7fe9ce052740 1 Processor -- start 2024-03-16T11:08:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93170 0x556f7fda1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9ce052740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9ce052740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41a4df0 con 0x556f7fd98cd0 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9ce052740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41a6260 con 0x556f7fd93b60 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93170 0x556f7fda1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9c89fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9c91fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93170 0x556f7fda1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50432/0 (socket says 172.21.15.78:50432) 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9c91fd640 1 -- 172.21.15.78:0/2084864698 learned_addr learned my addr 172.21.15.78:0/2084864698 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41a54c0 con 0x556f7fd93170 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9ce052740 10 client.0 _ll_register_callbacks cb 0x7fe9b40d2580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.380+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9c91fd640 1 -- 172.21.15.78:0/2084864698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 msgr2=0x556f7fda21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9c91fd640 1 -- 172.21.15.78:0/2084864698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 msgr2=0x556f7fda31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9c91fd640 1 -- 172.21.15.78:0/2084864698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f7fcbf9d0 con 0x556f7fd93170 2024-03-16T11:08:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9c99fe640 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9c91fd640 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93170 0x556f7fda1ca0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40a3a20 tx=0x7fe9b40a3d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=862b44fdecd24097 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.381+0000 7fe9acff9640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.382+0000 7fe9acff9640 1 -- 172.21.15.78:0/2084864698 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b407e060 con 0x556f7fd93170 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.382+0000 7fe9acff9640 1 -- 172.21.15.78:0/2084864698 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe9b409ac00 con 0x556f7fd93170 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.382+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9b409beb0 con 0x556f7fd93170 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.382+0000 7fe9acff9640 1 -- 172.21.15.78:0/2084864698 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe9b40ba050 con 0x556f7fd93170 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.382+0000 7fe9ce052740 10 client.25774 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.382+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556f7fda4870 con 0x556f7fd93170 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.382+0000 7fe9acff9640 1 -- 172.21.15.78:0/2084864698 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9b4003b30 con 0x556f7fd93170 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9acff9640 1 -- 172.21.15.78:0/2084864698 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe9b4083070 con 0x556f7fd93170 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9ce052740 10 client.25774 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9ce052740 10 client.25774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9ce052740 20 client.25774 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9ce052740 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fe0f970 0x556f7fe11d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9ce052740 4 client.25774 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe9b409beb0 con 0x556f7fe0f970 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fe0f970 0x556f7fe11d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.383+0000 7fe9c89fc640 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fe0f970 0x556f7fe11d60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5314d2d7ac254e2 server_cookie=4eb0b9dbae326343 in_seq=0 out_seq=0 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9acff9640 10 client.25774 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9acff9640 1 -- 172.21.15.78:0/2084864698 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fe9b409beb0 con 0x556f7fe0f970 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9acff9640 10 client.25774 handle_command_reply: tid=0 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 client.25774 shutdown 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93170 msgr2=0x556f7fda1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93170 0x556f7fda1ca0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40a3a20 tx=0x7fe9b40a3d50 comp rx=0 tx=0).stop 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 shutdown_connections 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556f7fe0f970 0x556f7fe11d60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f7fd98cd0 0x556f7fda31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556f7fd93b60 0x556f7fda21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 --2- 172.21.15.78:0/2084864698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556f7fd93170 0x556f7fda1ca0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40a3a20 tx=0x7fe9b40a3d50 comp rx=0 tx=0).stop 2024-03-16T11:08:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9acff9640 0 client.25774 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 >> 172.21.15.78:0/2084864698 conn(0x556f7fd8e6e0 msgr2=0x556f7fd898e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.384+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 shutdown_connections 2024-03-16T11:08:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9ce052740 1 -- 172.21.15.78:0/2084864698 wait complete. 2024-03-16T11:08:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9ce052740 20 client.25774 trim_cache size 0 max 16384 2024-03-16T11:08:08.387 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:08.388 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-16T11:08:08.388 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-16T11:08:08.388 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe9b00a28d0 msgr2=0x7fe9b00a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe9b00a28d0 0x7fe9b00a4d90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x556f7fd943c0 tx=0x7fe9bc00ba50 comp rx=0 tx=0).stop 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 msgr2=0x7fe9c4195600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 0x7fe9c4195600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400e9c0 tx=0x7fe9b400edc0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 shutdown_connections 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c41022d0 0x7fe9c419cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c4101910 0x7fe9c4195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe9b00a28d0 0x7fe9b00a4d90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 --2- 172.21.15.78:0/3638994877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c4100720 0x7fe9c4195600 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400e9c0 tx=0x7fe9b400edc0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.385+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 >> 172.21.15.78:0/3638994877 conn(0x7fe9c40fbc90 msgr2=0x7fe9c4104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.386+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 shutdown_connections 2024-03-16T11:08:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.386+0000 7fe9cb461640 1 -- 172.21.15.78:0/3638994877 wait complete. 2024-03-16T11:08:08.402 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T11:08:08.403 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:08:08.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2023141460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4101820 msgr2=0x7fc5f4101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2023141460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4101820 0x7fc5f4101c40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009860 tx=0x7fc5e402bb70 comp rx=0 tx=0).stop 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2023141460 shutdown_connections 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2023141460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f41033d0 0x7fc5f4109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2023141460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4102a10 0x7fc5f4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2023141460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4101820 0x7fc5f4101c40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2023141460 >> 172.21.15.78:0/2023141460 conn(0x7fc5f40fcd90 msgr2=0x7fc5f40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2023141460 shutdown_connections 2024-03-16T11:08:08.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2023141460 wait complete. 2024-03-16T11:08:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.559+0000 7fc5f9a7e640 1 Processor -- start 2024-03-16T11:08:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f9a7e640 1 -- start start 2024-03-16T11:08:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f9a7e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4101820 0x7fc5f4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f9a7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4102a10 0x7fc5f4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f9a7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f41033d0 0x7fc5f4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f9a7e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f410d320 con 0x7fc5f4101820 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f9a7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f410d1a0 con 0x7fc5f4102a10 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f9a7e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f410d4a0 con 0x7fc5f41033d0 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4101820 0x7fc5f4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4102a10 0x7fc5f4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4101820 0x7fc5f4075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41666/0 (socket says 172.21.15.78:41666) 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f37fe640 1 -- 172.21.15.78:0/2567956651 learned_addr learned my addr 172.21.15.78:0/2567956651 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f3fff640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f41033d0 0x7fc5f4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f37fe640 1 -- 172.21.15.78:0/2567956651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f41033d0 msgr2=0x7fc5f4074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.560+0000 7fc5f37fe640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f41033d0 0x7fc5f4074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f37fe640 1 -- 172.21.15.78:0/2567956651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4102a10 msgr2=0x7fc5f4073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f37fe640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4102a10 0x7fc5f4073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f37fe640 1 -- 172.21.15.78:0/2567956651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f40749a0 con 0x7fc5f4101820 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f2ffd640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4102a10 0x7fc5f4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f37fe640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4101820 0x7fc5f4075590 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4005940 tx=0x7fc5e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f0ff9640 1 -- 172.21.15.78:0/2567956651 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4041020 con 0x7fc5f4101820 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f41a1b20 con 0x7fc5f4101820 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f0ff9640 1 -- 172.21.15.78:0/2567956651 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5e40035f0 con 0x7fc5f4101820 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f0ff9640 1 -- 172.21.15.78:0/2567956651 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4039040 con 0x7fc5f4101820 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.561+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f41a20c0 con 0x7fc5f4101820 2024-03-16T11:08:08.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.562+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f4108cb0 con 0x7fc5f4101820 2024-03-16T11:08:08.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.565+0000 7fc5f0ff9640 1 -- 172.21.15.78:0/2567956651 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc5e4044020 con 0x7fc5f4101820 2024-03-16T11:08:08.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.566+0000 7fc5f0ff9640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5dc0a2800 0x7fc5dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:08.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.566+0000 7fc5f0ff9640 1 -- 172.21.15.78:0/2567956651 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc5e40f05a0 con 0x7fc5f4101820 2024-03-16T11:08:08.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.566+0000 7fc5f2ffd640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5dc0a2800 0x7fc5dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:08.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.567+0000 7fc5f2ffd640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5dc0a2800 0x7fc5dc0a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4075300 tx=0x7fc5e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:08.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.570+0000 7fc5f0ff9640 1 -- 172.21.15.78:0/2567956651 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc5e404b070 con 0x7fc5f4101820 2024-03-16T11:08:08.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.945+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5f402d1b0 con 0x7fc5f4101820 2024-03-16T11:08:08.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.947+0000 7fc5f0ff9640 1 -- 172.21.15.78:0/2567956651 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7fc5e40bd600 con 0x7fc5f4101820 2024-03-16T11:08:08.948 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:08:08.948 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:08:08.948 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5dc0a2800 msgr2=0x7fc5dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5dc0a2800 0x7fc5dc0a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4075300 tx=0x7fc5e8005f90 comp rx=0 tx=0).stop 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4101820 msgr2=0x7fc5f4075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4101820 0x7fc5f4075590 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4005940 tx=0x7fc5e40042f0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 shutdown_connections 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc5dc0a2800 0x7fc5dc0a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f41033d0 0x7fc5f4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f4102a10 0x7fc5f4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 --2- 172.21.15.78:0/2567956651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5f4101820 0x7fc5f4075590 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 >> 172.21.15.78:0/2567956651 conn(0x7fc5f40fcd90 msgr2=0x7fc5f4105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:08.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.950+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 shutdown_connections 2024-03-16T11:08:08.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:08.951+0000 7fc5f9a7e640 1 -- 172.21.15.78:0/2567956651 wait complete. 2024-03-16T11:08:08.968 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 -- 172.21.15.78:0/1697167372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 msgr2=0x7f0e3c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 --2- 172.21.15.78:0/1697167372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 0x7f0e3c108b40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009be0 tx=0x7f0e3402bef0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 -- 172.21.15.78:0/1697167372 shutdown_connections 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 --2- 172.21.15.78:0/1697167372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 0x7f0e3c108b40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 --2- 172.21.15.78:0/1697167372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1018f0 0x7f0e3c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 --2- 172.21.15.78:0/1697167372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 0x7f0e3c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 -- 172.21.15.78:0/1697167372 >> 172.21.15.78:0/1697167372 conn(0x7f0e3c0fbc90 msgr2=0x7f0e3c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 -- 172.21.15.78:0/1697167372 shutdown_connections 2024-03-16T11:08:09.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.124+0000 7f0e43e36640 1 -- 172.21.15.78:0/1697167372 wait complete. 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.125+0000 7f0e43e36640 1 Processor -- start 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e43e36640 1 -- start start 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e43e36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 0x7f0e3c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e43e36640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1018f0 0x7f0e3c196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e43e36640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 0x7f0e3c19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e43e36640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10c260 con 0x7f0e3c1018f0 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e43e36640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10c0e0 con 0x7f0e3c100700 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e43e36640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10c3e0 con 0x7f0e3c1022b0 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e41bd2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 0x7f0e3c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e41bd2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 0x7f0e3c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36620/0 (socket says 172.21.15.78:36620) 2024-03-16T11:08:09.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e41bd2640 1 -- 172.21.15.78:0/2541230134 learned_addr learned my addr 172.21.15.78:0/2541230134 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e423d3640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 0x7f0e3c19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.126+0000 7f0e413d1640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1018f0 0x7f0e3c196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.127+0000 7f0e41bd2640 1 -- 172.21.15.78:0/2541230134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 msgr2=0x7f0e3c19de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.127+0000 7f0e41bd2640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 0x7f0e3c19de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.127+0000 7f0e41bd2640 1 -- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1018f0 msgr2=0x7f0e3c196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.127+0000 7f0e41bd2640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1018f0 0x7f0e3c196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.127+0000 7f0e41bd2640 1 -- 172.21.15.78:0/2541230134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3c19e4f0 con 0x7f0e3c100700 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.127+0000 7f0e41bd2640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 0x7f0e3c1968d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800d770 tx=0x7f0e3800db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.127+0000 7f0e32ffd640 1 -- 172.21.15.78:0/2541230134 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38018070 con 0x7f0e3c100700 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.128+0000 7f0e32ffd640 1 -- 172.21.15.78:0/2541230134 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e38009cd0 con 0x7f0e3c100700 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.128+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3c069ba0 con 0x7f0e3c100700 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.128+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3c06a140 con 0x7f0e3c100700 2024-03-16T11:08:09.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.128+0000 7f0e32ffd640 1 -- 172.21.15.78:0/2541230134 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38012dd0 con 0x7f0e3c100700 2024-03-16T11:08:09.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.131+0000 7f0e32ffd640 1 -- 172.21.15.78:0/2541230134 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0e38009820 con 0x7f0e3c100700 2024-03-16T11:08:09.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.131+0000 7f0e46a24740 1 Processor -- start 2024-03-16T11:08:09.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e46a24740 1 -- start start 2024-03-16T11:08:09.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69d850 0x55a7fa69dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e46a24740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa602490 con 0x55a7fa69e240 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e46a24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa5aac90 con 0x55a7fa6a33b0 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e46a24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa5fe530 con 0x55a7fa69d850 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69d850 0x55a7fa69dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69d850 0x55a7fa69dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50470/0 (socket says 172.21.15.78:50470) 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 -- 172.21.15.78:0/4003589025 learned_addr learned my addr 172.21.15.78:0/4003589025 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e413d1640 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e41bd2640 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 -- 172.21.15.78:0/4003589025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 msgr2=0x55a7fa6a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6a5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 -- 172.21.15.78:0/4003589025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 msgr2=0x55a7fa6a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e423d3640 1 -- 172.21.15.78:0/4003589025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fa665500 con 0x55a7fa69d850 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.132+0000 7f0e32ffd640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e280a2880 0x7f0e280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.133+0000 7f0e41bd2640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e280a2880 0x7f0e280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.133+0000 7f0e423d3640 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69d850 0x55a7fa69dc70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0e3402c500 tx=0x7f0e340098f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9022c5d6754c52e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.133+0000 7f0e27fff640 1 -- 172.21.15.78:0/4003589025 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34043070 con 0x55a7fa69d850 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.133+0000 7f0e27fff640 1 -- 172.21.15.78:0/4003589025 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e3402c890 con 0x55a7fa69d850 2024-03-16T11:08:09.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.133+0000 7f0e27fff640 1 -- 172.21.15.78:0/4003589025 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e340349d0 con 0x55a7fa69d850 2024-03-16T11:08:09.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.133+0000 7f0e32ffd640 1 -- 172.21.15.78:0/2541230134 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0e380d24f0 con 0x7f0e3c100700 2024-03-16T11:08:09.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 -- 172.21.15.78:0/4003589025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69d850 msgr2=0x55a7fa69dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69d850 0x55a7fa69dc70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0e3402c500 tx=0x7f0e340098f0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e41bd2640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e280a2880 0x7f0e280a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0e380998f0 tx=0x7f0e38021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 -- 172.21.15.78:0/4003589025 shutdown_connections 2024-03-16T11:08:09.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 --2- 172.21.15.78:0/4003589025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69d850 0x55a7fa69dc70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 -- 172.21.15.78:0/4003589025 >> 172.21.15.78:0/4003589025 conn(0x55a7fa698dc0 msgr2=0x55a7fa6991c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.134+0000 7f0e46a24740 1 -- 172.21.15.78:0/4003589025 shutdown_connections 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.135+0000 7f0e46a24740 1 -- 172.21.15.78:0/4003589025 wait complete. 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.135+0000 7f0e46a24740 1 -- start start 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.135+0000 7f0e46a24740 1 Processor -- start 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69d850 0x55a7fa6a7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6af310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e46a24740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa6a6300 con 0x55a7fa69e240 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e46a24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa5be230 con 0x55a7fa69d850 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e46a24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa6a6480 con 0x55a7fa6a33b0 2024-03-16T11:08:09.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e46a24740 10 client.0 _ll_register_callbacks cb 0x55a7fa5fa010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69d850 0x55a7fa6a7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69d850 0x55a7fa6a7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36626/0 (socket says 172.21.15.78:36626) 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 -- 172.21.15.78:0/1943060336 learned_addr learned my addr 172.21.15.78:0/1943060336 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e413d1640 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 -- 172.21.15.78:0/1943060336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 msgr2=0x55a7fa6af310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6af310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 -- 172.21.15.78:0/1943060336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 msgr2=0x55a7fa6a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.136+0000 7f0e423d3640 1 -- 172.21.15.78:0/1943060336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fa5cd100 con 0x55a7fa69d850 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.137+0000 7f0e423d3640 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69d850 0x55a7fa6a7d90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0e3402f020 tx=0x7f0e34009960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4744023666cbd6f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.137+0000 7f0e257fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.137+0000 7f0e257fa640 1 -- 172.21.15.78:0/1943060336 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34043040 con 0x55a7fa69d850 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.137+0000 7f0e257fa640 1 -- 172.21.15.78:0/1943060336 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0e34004b40 con 0x55a7fa69d850 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.137+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e3402c890 con 0x55a7fa69d850 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e257fa640 1 -- 172.21.15.78:0/1943060336 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0e34004d40 con 0x55a7fa69d850 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e46a24740 10 client.25814 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a7fa3163f0 con 0x55a7fa69d850 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e257fa640 1 -- 172.21.15.78:0/1943060336 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e3402c890 con 0x55a7fa69d850 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e257fa640 1 -- 172.21.15.78:0/1943060336 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0e340484e0 con 0x55a7fa69d850 2024-03-16T11:08:09.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e46a24740 10 client.25814 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e46a24740 10 client.25814 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.138+0000 7f0e46a24740 20 client.25814 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.139+0000 7f0e46a24740 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa6b3800 0x55a7fa6b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.139+0000 7f0e46a24740 4 client.25814 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.139+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7fa60f9d0 con 0x55a7fa6b3800 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.139+0000 7f0e413d1640 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa6b3800 0x55a7fa6b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.139+0000 7f0e257fa640 10 client.25814 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.139+0000 7f0e413d1640 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa6b3800 0x55a7fa6b5bf0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b9a1d681799ba25 server_cookie=63a7bdee24af5f43 in_seq=0 out_seq=0 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.140+0000 7f0e257fa640 1 -- 172.21.15.78:0/1943060336 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7fa60f9d0 con 0x55a7fa6b3800 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e257fa640 10 client.25814 handle_command_reply: tid=0 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 client.25814 shutdown 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69d850 msgr2=0x55a7fa6a7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69d850 0x55a7fa6a7d90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0e3402f020 tx=0x7f0e34009960 comp rx=0 tx=0).stop 2024-03-16T11:08:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 shutdown_connections 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa6b3800 0x55a7fa6b5bf0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69e240 0x55a7fa6a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 --2- 172.21.15.78:0/1943060336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69d850 0x55a7fa6a7d90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 >> 172.21.15.78:0/1943060336 conn(0x55a7fa698dc0 msgr2=0x55a7fa697fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e257fa640 0 client.25814 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 shutdown_connections 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 1 -- 172.21.15.78:0/1943060336 wait complete. 2024-03-16T11:08:09.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.141+0000 7f0e46a24740 20 client.25814 trim_cache size 0 max 16384 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.161+0000 7f0e46a24740 1 Processor -- start 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e46a24740 1 -- start start 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6a8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69e240 0x55a7fa6af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ba350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e46a24740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0e20010d90 con 0x55a7fa69d850 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e46a24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2000e660 con 0x55a7fa69e240 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e46a24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2000c490 con 0x55a7fa6a33b0 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e413d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69e240 0x55a7fa6af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e413d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69e240 0x55a7fa6af2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36642/0 (socket says 172.21.15.78:36642) 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e413d1640 1 -- 172.21.15.78:0/3541619323 learned_addr learned my addr 172.21.15.78:0/3541619323 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e423d3640 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6a8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e423d3640 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ba350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.162+0000 7f0e413d1640 1 -- 172.21.15.78:0/3541619323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 msgr2=0x55a7fa6ba350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e413d1640 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ba350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e413d1640 1 -- 172.21.15.78:0/3541619323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 msgr2=0x55a7fa6a8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e413d1640 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6a8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e413d1640 1 -- 172.21.15.78:0/3541619323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fa3e07d0 con 0x55a7fa69e240 2024-03-16T11:08:09.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e413d1640 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69e240 0x55a7fa6af2a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009d10 tx=0x7f0e2c01ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76c7964c779c59b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e277fe640 1 -- 172.21.15.78:0/3541619323 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c01e430 con 0x55a7fa69e240 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e277fe640 1 -- 172.21.15.78:0/3541619323 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e2c01e5d0 con 0x55a7fa69e240 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.163+0000 7f0e277fe640 1 -- 172.21.15.78:0/3541619323 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c005020 con 0x55a7fa69e240 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.164+0000 7f0e46a24740 1 -- 172.21.15.78:0/3541619323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69e240 msgr2=0x55a7fa6af2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.164+0000 7f0e46a24740 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69e240 0x55a7fa6af2a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009d10 tx=0x7f0e2c01ef60 comp rx=0 tx=0).stop 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 -- 172.21.15.78:0/3541619323 shutdown_connections 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ba350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa69e240 0x55a7fa6af2a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 --2- 172.21.15.78:0/3541619323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6a8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 -- 172.21.15.78:0/3541619323 >> 172.21.15.78:0/3541619323 conn(0x55a7fa698dc0 msgr2=0x55a7fa69c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 -- 172.21.15.78:0/3541619323 shutdown_connections 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 -- 172.21.15.78:0/3541619323 wait complete. 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 -- start start 2024-03-16T11:08:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.165+0000 7f0e46a24740 1 Processor -- start 2024-03-16T11:08:09.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69e240 0x55a7fa6ad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e46a24740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e46a24740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2000cb20 con 0x55a7fa69d850 2024-03-16T11:08:09.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e46a24740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2000df90 con 0x55a7fa6a33b0 2024-03-16T11:08:09.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e46a24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2000d1f0 con 0x55a7fa69e240 2024-03-16T11:08:09.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e46a24740 10 client.0 _ll_register_callbacks cb 0x7f0e34009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:09.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e413d1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69e240 0x55a7fa6ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e423d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e423d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6acee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41720/0 (socket says 172.21.15.78:41720) 2024-03-16T11:08:09.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e423d3640 1 -- 172.21.15.78:0/196531768 learned_addr learned my addr 172.21.15.78:0/196531768 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.167+0000 7f0e423d3640 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.168+0000 7f0e423d3640 1 -- 172.21.15.78:0/196531768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69e240 msgr2=0x55a7fa6ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.168+0000 7f0e423d3640 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69e240 0x55a7fa6ad420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.168+0000 7f0e423d3640 1 -- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 msgr2=0x55a7fa6ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.168+0000 7f0e423d3640 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.168+0000 7f0e423d3640 1 -- 172.21.15.78:0/196531768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fa6aea20 con 0x55a7fa69d850 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.169+0000 7f0e423d3640 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6acee0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009be0 tx=0x7f0e3404a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3f8b333a9a0460 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.169+0000 7f0e257fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:09.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.170+0000 7f0e257fa640 1 -- 172.21.15.78:0/196531768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34043040 con 0x55a7fa69d850 2024-03-16T11:08:09.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.171+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e2c01e5d0 con 0x55a7fa69d850 2024-03-16T11:08:09.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.170+0000 7f0e257fa640 1 -- 172.21.15.78:0/196531768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0e34048450 con 0x55a7fa69d850 2024-03-16T11:08:09.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.171+0000 7f0e257fa640 1 -- 172.21.15.78:0/196531768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0e34048650 con 0x55a7fa69d850 2024-03-16T11:08:09.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.171+0000 7f0e46a24740 10 client.15990 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:09.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.171+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a7fa722680 con 0x55a7fa69d850 2024-03-16T11:08:09.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e257fa640 1 -- 172.21.15.78:0/196531768 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e2c01e5d0 con 0x55a7fa69d850 2024-03-16T11:08:09.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e257fa640 1 -- 172.21.15.78:0/196531768 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0e340664e0 con 0x55a7fa69d850 2024-03-16T11:08:09.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e46a24740 10 client.15990 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:09.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e46a24740 10 client.15990 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:08:09.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e46a24740 20 client.15990 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:09.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e46a24740 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa722e70 0x55a7fa725290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e46a24740 4 client.15990 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:08:09.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.172+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7fa609d20 con 0x55a7fa722e70 2024-03-16T11:08:09.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.173+0000 7f0e413d1640 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa722e70 0x55a7fa725290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.173+0000 7f0e257fa640 10 client.15990 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.173+0000 7f0e413d1640 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa722e70 0x55a7fa725290 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4649f6a22ad1f76 server_cookie=ef05e5ae1c40359c in_seq=0 out_seq=0 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e257fa640 1 -- 172.21.15.78:0/196531768 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7fa609d20 con 0x55a7fa722e70 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e257fa640 10 client.15990 handle_command_reply: tid=0 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 client.15990 shutdown 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 msgr2=0x55a7fa6acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6acee0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009be0 tx=0x7f0e3404a040 comp rx=0 tx=0).stop 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 shutdown_connections 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55a7fa722e70 0x55a7fa725290 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7fa6a33b0 0x55a7fa6ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7fa69e240 0x55a7fa6ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 --2- 172.21.15.78:0/196531768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a7fa69d850 0x55a7fa6acee0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 >> 172.21.15.78:0/196531768 conn(0x55a7fa698dc0 msgr2=0x55a7fa69c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e257fa640 0 client.15990 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 shutdown_connections 2024-03-16T11:08:09.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 1 -- 172.21.15.78:0/196531768 wait complete. 2024-03-16T11:08:09.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.174+0000 7f0e46a24740 20 client.15990 trim_cache size 0 max 16384 2024-03-16T11:08:09.176 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e280a2880 msgr2=0x7f0e280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e280a2880 0x7f0e280a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0e380998f0 tx=0x7f0e38021000 comp rx=0 tx=0).stop 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 msgr2=0x7f0e3c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 0x7f0e3c1968d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800d770 tx=0x7f0e3800db70 comp rx=0 tx=0).stop 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 shutdown_connections 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0e280a2880 0x7f0e280a4d40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e3c1022b0 0x7f0e3c19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1018f0 0x7f0e3c196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 --2- 172.21.15.78:0/2541230134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e3c100700 0x7f0e3c1968d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 >> 172.21.15.78:0/2541230134 conn(0x7f0e3c0fbc90 msgr2=0x7f0e3c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 shutdown_connections 2024-03-16T11:08:09.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.175+0000 7f0e43e36640 1 -- 172.21.15.78:0/2541230134 wait complete. 2024-03-16T11:08:09.192 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T11:08:09.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:09 smithi078 ceph-mon[32235]: pgmap v413: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:09.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:09 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2567956651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 -- 172.21.15.78:0/3248196959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481009e0 msgr2=0x7efe48100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 --2- 172.21.15.78:0/3248196959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481009e0 0x7efe48100e60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7efe38009900 tx=0x7efe3802bc50 comp rx=0 tx=0).stop 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 -- 172.21.15.78:0/3248196959 shutdown_connections 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 --2- 172.21.15.78:0/3248196959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe481013a0 0x7efe48108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 --2- 172.21.15.78:0/3248196959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481009e0 0x7efe48100e60 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 --2- 172.21.15.78:0/3248196959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe48101cb0 0x7efe481020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 -- 172.21.15.78:0/3248196959 >> 172.21.15.78:0/3248196959 conn(0x7efe480fb410 msgr2=0x7efe480fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 -- 172.21.15.78:0/3248196959 shutdown_connections 2024-03-16T11:08:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.351+0000 7efe4f40b640 1 -- 172.21.15.78:0/3248196959 wait complete. 2024-03-16T11:08:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.353+0000 7efe4f40b640 1 Processor -- start 2024-03-16T11:08:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.353+0000 7efe4f40b640 1 -- start start 2024-03-16T11:08:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.353+0000 7efe4f40b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe481009e0 0x7efe481969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.353+0000 7efe4f40b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481013a0 0x7efe48196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe481009e0 0x7efe481969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe481009e0 0x7efe481969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36670/0 (socket says 172.21.15.78:36670) 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 -- 172.21.15.78:0/1055071960 learned_addr learned my addr 172.21.15.78:0/1055071960 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4f40b640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48101cb0 0x7efe4819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481013a0 0x7efe48196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810c340 con 0x7efe481013a0 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810c1c0 con 0x7efe481009e0 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810c4c0 con 0x7efe48101cb0 2024-03-16T11:08:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48101cb0 0x7efe4819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 -- 172.21.15.78:0/1055071960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48101cb0 msgr2=0x7efe4819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48101cb0 0x7efe4819dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 -- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481013a0 msgr2=0x7efe48196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481013a0 0x7efe48196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.354+0000 7efe4d1a7640 1 -- 172.21.15.78:0/1055071960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe4819e610 con 0x7efe481009e0 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.355+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe481009e0 0x7efe481969f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe4400b470 tx=0x7efe4400b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.355+0000 7efe3e7fc640 1 -- 172.21.15.78:0/1055071960 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44015020 con 0x7efe481009e0 2024-03-16T11:08:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.355+0000 7efe3e7fc640 1 -- 172.21.15.78:0/1055071960 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe44011d40 con 0x7efe481009e0 2024-03-16T11:08:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.355+0000 7efe3e7fc640 1 -- 172.21.15.78:0/1055071960 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4400f340 con 0x7efe481009e0 2024-03-16T11:08:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.355+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe4819e900 con 0x7efe481009e0 2024-03-16T11:08:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.355+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe4819eef0 con 0x7efe481009e0 2024-03-16T11:08:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.358+0000 7efe3e7fc640 1 -- 172.21.15.78:0/1055071960 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7efe44018020 con 0x7efe481009e0 2024-03-16T11:08:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 Processor -- start 2024-03-16T11:08:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe3e7fc640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe340a2950 0x7efe340a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 -- start start 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe340a2950 0x7efe340a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf0110c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf0115d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0118840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555bf0075490 con 0x555bf0111240 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555bf001dc90 con 0x555bf01163b0 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe51ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555bf0071530 con 0x555bf0110850 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4c9a6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf0115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4c9a6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf0115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41744/0 (socket says 172.21.15.78:41744) 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4c9a6640 1 -- 172.21.15.78:0/1601389568 learned_addr learned my addr 172.21.15.78:0/1601389568 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0118840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf0110c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe3e7fc640 1 -- 172.21.15.78:0/1055071960 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7efe440ce3d0 con 0x7efe481009e0 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4c9a6640 1 -- 172.21.15.78:0/1601389568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 msgr2=0x555bf0110c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf0110c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4c9a6640 1 -- 172.21.15.78:0/1601389568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 msgr2=0x555bf0118840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.359+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0118840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.360+0000 7efe4c9a6640 1 -- 172.21.15.78:0/1601389568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bf00d8500 con 0x555bf0111240 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.360+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf0110c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.360+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe340a2950 0x7efe340a4e10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efe40005fd0 tx=0x7efe40011bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.360+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf0115d40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efe38009230 tx=0x7efe38005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f24ce3136290b89 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.360+0000 7efe337fe640 1 -- 172.21.15.78:0/1601389568 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3802ca00 con 0x555bf0111240 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.360+0000 7efe337fe640 1 -- 172.21.15.78:0/1601389568 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3802cba0 con 0x555bf0111240 2024-03-16T11:08:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.360+0000 7efe337fe640 1 -- 172.21.15.78:0/1601389568 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38040930 con 0x555bf0111240 2024-03-16T11:08:09.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.361+0000 7efe51ff5740 1 -- 172.21.15.78:0/1601389568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 msgr2=0x555bf0115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.361+0000 7efe51ff5740 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf0115d40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efe38009230 tx=0x7efe38005e70 comp rx=0 tx=0).stop 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.361+0000 7efe51ff5740 1 -- 172.21.15.78:0/1601389568 shutdown_connections 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.361+0000 7efe51ff5740 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0118840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.361+0000 7efe51ff5740 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf0115d40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.361+0000 7efe51ff5740 1 --2- 172.21.15.78:0/1601389568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf0110c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.361+0000 7efe51ff5740 1 -- 172.21.15.78:0/1601389568 >> 172.21.15.78:0/1601389568 conn(0x555bf010bdc0 msgr2=0x555bf010c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.362+0000 7efe51ff5740 1 -- 172.21.15.78:0/1601389568 shutdown_connections 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.362+0000 7efe51ff5740 1 -- 172.21.15.78:0/1601389568 wait complete. 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.362+0000 7efe51ff5740 1 -- start start 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.362+0000 7efe51ff5740 1 Processor -- start 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.363+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf011d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.363+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf011fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.363+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0122070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe51ff5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555bf01191c0 con 0x555bf0111240 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe51ff5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555bf0031230 con 0x555bf01163b0 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4d1a7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf011d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe51ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555bf0119340 con 0x555bf0110850 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4d1a7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf011d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50544/0 (socket says 172.21.15.78:50544) 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4d1a7640 1 -- 172.21.15.78:0/211511328 learned_addr learned my addr 172.21.15.78:0/211511328 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf011fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe51ff5740 10 client.0 _ll_register_callbacks cb 0x555bf006d010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0122070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4c9a6640 1 -- 172.21.15.78:0/211511328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 msgr2=0x555bf011d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf011d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4c9a6640 1 -- 172.21.15.78:0/211511328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 msgr2=0x555bf0122070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0122070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4c9a6640 1 -- 172.21.15.78:0/211511328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf0040100 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0122070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf011fa80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x555bf010cd30 tx=0x7efe38040900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c54aedf85f85069 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.364+0000 7efe30ff9640 1 -- 172.21.15.78:0/211511328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38051050 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe30ff9640 1 -- 172.21.15.78:0/211511328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7efe380035f0 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe3802cba0 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe30ff9640 1 -- 172.21.15.78:0/211511328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efe3802ca00 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 10 client.16002 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555befd893f0 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe30ff9640 1 -- 172.21.15.78:0/211511328 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3802cba0 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe30ff9640 1 -- 172.21.15.78:0/211511328 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7efe3804b630 con 0x555bf0111240 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 10 client.16002 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 10 client.16002 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:08:09.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 20 client.16002 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:09.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0126790 0x555bf0128b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 4 client.16002 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:09.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.365+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bf00829d0 con 0x555bf0126790 2024-03-16T11:08:09.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.366+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0126790 0x555bf0128b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.366+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0126790 0x555bf0128b80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73c913175ce64a4e server_cookie=1f758c56becb7812 in_seq=0 out_seq=0 2024-03-16T11:08:09.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.366+0000 7efe30ff9640 10 client.16002 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.367+0000 7efe30ff9640 1 -- 172.21.15.78:0/211511328 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555bf00829d0 con 0x555bf0126790 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.367+0000 7efe30ff9640 10 client.16002 handle_command_reply: tid=0 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.367+0000 7efe51ff5740 1 client.16002 shutdown 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.367+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 msgr2=0x555bf011fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.367+0000 7efe51ff5740 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf011fa80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x555bf010cd30 tx=0x7efe38040900 comp rx=0 tx=0).stop 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 shutdown_connections 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0126790 0x555bf0128b80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf01163b0 0x555bf0122070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe30ff9640 0 client.16002 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:09.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0111240 0x555bf011fa80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 --2- 172.21.15.78:0/211511328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0110850 0x555bf011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 >> 172.21.15.78:0/211511328 conn(0x555bf010bdc0 msgr2=0x555bf010afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 shutdown_connections 2024-03-16T11:08:09.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 1 -- 172.21.15.78:0/211511328 wait complete. 2024-03-16T11:08:09.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.368+0000 7efe51ff5740 20 client.16002 trim_cache size 0 max 16384 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.386+0000 7efe51ff5740 1 Processor -- start 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.386+0000 7efe51ff5740 1 -- start start 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0110850 0x555bf0114bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0111240 0x555bf011a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf01163b0 0x555bf010ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe51ff5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a9380 con 0x555bf01163b0 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe51ff5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a6c50 con 0x555bf0110850 2024-03-16T11:08:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe51ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a4a80 con 0x555bf0111240 2024-03-16T11:08:09.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf01163b0 0x555bf010ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf01163b0 0x555bf010ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41758/0 (socket says 172.21.15.78:41758) 2024-03-16T11:08:09.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 -- 172.21.15.78:0/683909483 learned_addr learned my addr 172.21.15.78:0/683909483 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4c9a6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0111240 0x555bf011a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0110850 0x555bf0114bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 -- 172.21.15.78:0/683909483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0111240 msgr2=0x555bf011a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0111240 0x555bf011a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 -- 172.21.15.78:0/683909483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0110850 msgr2=0x555bf0114bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0110850 0x555bf0114bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 -- 172.21.15.78:0/683909483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555befe537d0 con 0x555bf01163b0 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.387+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf01163b0 0x555bf010ae60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7efe4000fcb0 tx=0x7efe4001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60670cde99613ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.388+0000 7efe32ffd640 1 -- 172.21.15.78:0/683909483 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4001a040 con 0x555bf01163b0 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.388+0000 7efe32ffd640 1 -- 172.21.15.78:0/683909483 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe40018040 con 0x555bf01163b0 2024-03-16T11:08:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.388+0000 7efe32ffd640 1 -- 172.21.15.78:0/683909483 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe400283d0 con 0x555bf01163b0 2024-03-16T11:08:09.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 -- 172.21.15.78:0/683909483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf01163b0 msgr2=0x555bf010ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf01163b0 0x555bf010ae60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7efe4000fcb0 tx=0x7efe4001f000 comp rx=0 tx=0).stop 2024-03-16T11:08:09.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 -- 172.21.15.78:0/683909483 shutdown_connections 2024-03-16T11:08:09.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf01163b0 0x555bf010ae60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf0111240 0x555bf011a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 --2- 172.21.15.78:0/683909483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0110850 0x555bf0114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 -- 172.21.15.78:0/683909483 >> 172.21.15.78:0/683909483 conn(0x555bf010bdc0 msgr2=0x555bf010a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 -- 172.21.15.78:0/683909483 shutdown_connections 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 -- 172.21.15.78:0/683909483 wait complete. 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.389+0000 7efe51ff5740 1 -- start start 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 1 Processor -- start 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0110850 0x555bf01376f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0111240 0x555bf0137c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf01163b0 0x555bf0138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a5110 con 0x555bf0110850 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a6580 con 0x555bf0111240 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a57e0 con 0x555bf01163b0 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe51ff5740 10 client.0 _ll_register_callbacks cb 0x7efe3802cd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.390+0000 7efe4d1a7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0110850 0x555bf01376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0110850 0x555bf01376f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41764/0 (socket says 172.21.15.78:41764) 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 -- 172.21.15.78:0/3074304126 learned_addr learned my addr 172.21.15.78:0/3074304126 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d9a8640 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf01163b0 0x555bf0138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0111240 0x555bf0137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 -- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf01163b0 msgr2=0x555bf0138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf01163b0 0x555bf0138bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 -- 172.21.15.78:0/3074304126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0111240 msgr2=0x555bf0137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0111240 0x555bf0137c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 -- 172.21.15.78:0/3074304126 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf01391e0 con 0x555bf0110850 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe4d1a7640 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0110850 0x555bf01376f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7efe44000c00 tx=0x7efe440cbe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6fae8c3c9cc94 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe30ff9640 1 -- 172.21.15.78:0/3074304126 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4409ad30 con 0x555bf0110850 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe30ff9640 1 -- 172.21.15.78:0/3074304126 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7efe440973a0 con 0x555bf0110850 2024-03-16T11:08:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.391+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe40018040 con 0x555bf0110850 2024-03-16T11:08:09.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.392+0000 7efe30ff9640 1 -- 172.21.15.78:0/3074304126 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efe4409ee80 con 0x555bf0110850 2024-03-16T11:08:09.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.392+0000 7efe51ff5740 10 client.16008 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:09.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.392+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555bf01950a0 con 0x555bf0110850 2024-03-16T11:08:09.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.392+0000 7efe30ff9640 1 -- 172.21.15.78:0/3074304126 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe40018040 con 0x555bf0110850 2024-03-16T11:08:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe30ff9640 1 -- 172.21.15.78:0/3074304126 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7efe4404e670 con 0x555bf0110850 2024-03-16T11:08:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe51ff5740 10 client.16008 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe51ff5740 10 client.16008 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:08:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe51ff5740 20 client.16008 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe51ff5740 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0195a20 0x555bf0197e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe51ff5740 4 client.16008 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555bf007cd20 con 0x555bf0195a20 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0195a20 0x555bf0197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe30ff9640 10 client.16008 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.393+0000 7efe4c9a6640 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0195a20 0x555bf0197e40 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=45910d70d5cfc1f9 server_cookie=1f408df6114e8e77 in_seq=0 out_seq=0 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe30ff9640 1 -- 172.21.15.78:0/3074304126 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555bf007cd20 con 0x555bf0195a20 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe30ff9640 10 client.16008 handle_command_reply: tid=0 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 client.16008 shutdown 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0110850 msgr2=0x555bf01376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0110850 0x555bf01376f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7efe44000c00 tx=0x7efe440cbe60 comp rx=0 tx=0).stop 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 shutdown_connections 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x555bf0195a20 0x555bf0197e40 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bf01163b0 0x555bf0138bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555bf0111240 0x555bf0137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 --2- 172.21.15.78:0/3074304126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555bf0110850 0x555bf01376f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 >> 172.21.15.78:0/3074304126 conn(0x555bf010bdc0 msgr2=0x555bf010a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe30ff9640 0 client.16008 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:09.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 shutdown_connections 2024-03-16T11:08:09.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.394+0000 7efe51ff5740 1 -- 172.21.15.78:0/3074304126 wait complete. 2024-03-16T11:08:09.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.395+0000 7efe51ff5740 20 client.16008 trim_cache size 0 max 16384 2024-03-16T11:08:09.398 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.395+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe340a2950 msgr2=0x7efe340a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.395+0000 7efe4f40b640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe340a2950 0x7efe340a4e10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efe40005fd0 tx=0x7efe40011bc0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.395+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe481009e0 msgr2=0x7efe481969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.395+0000 7efe4f40b640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe481009e0 0x7efe481969f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe4400b470 tx=0x7efe4400b930 comp rx=0 tx=0).stop 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 shutdown_connections 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7efe340a2950 0x7efe340a4e10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48101cb0 0x7efe4819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe481013a0 0x7efe48196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 --2- 172.21.15.78:0/1055071960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe481009e0 0x7efe481969f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 >> 172.21.15.78:0/1055071960 conn(0x7efe480fb410 msgr2=0x7efe48104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 shutdown_connections 2024-03-16T11:08:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.396+0000 7efe4f40b640 1 -- 172.21.15.78:0/1055071960 wait complete. 2024-03-16T11:08:09.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T11:08:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:09 smithi098 ceph-mon[35848]: pgmap v413: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:09 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2567956651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:08:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:09 smithi153 ceph-mon[35521]: pgmap v413: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:09 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2567956651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:08:09.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.608+0000 7f66b33f3640 1 -- 172.21.15.78:0/129321479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 msgr2=0x7f66ac102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.608+0000 7f66b33f3640 1 --2- 172.21.15.78:0/129321479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 0x7f66ac102e80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009860 tx=0x7f66a802bb90 comp rx=0 tx=0).stop 2024-03-16T11:08:09.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.608+0000 7f66b33f3640 1 -- 172.21.15.78:0/129321479 shutdown_connections 2024-03-16T11:08:09.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.608+0000 7f66b33f3640 1 --2- 172.21.15.78:0/129321479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ac1033c0 0x7f66ac109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.608+0000 7f66b33f3640 1 --2- 172.21.15.78:0/129321479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 0x7f66ac102e80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.608+0000 7f66b33f3640 1 --2- 172.21.15.78:0/129321479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66ac101810 0x7f66ac101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.608+0000 7f66b33f3640 1 -- 172.21.15.78:0/129321479 >> 172.21.15.78:0/129321479 conn(0x7f66ac0fcd80 msgr2=0x7f66ac0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.609+0000 7f66b33f3640 1 -- 172.21.15.78:0/129321479 shutdown_connections 2024-03-16T11:08:09.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.609+0000 7f66b33f3640 1 -- 172.21.15.78:0/129321479 wait complete. 2024-03-16T11:08:09.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 Processor -- start 2024-03-16T11:08:09.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 -- start start 2024-03-16T11:08:09.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ac101810 0x7f66ac075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 0x7f66ac073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66ac1033c0 0x7f66ac074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b098e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 0x7f66ac073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b098e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 0x7f66ac073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36718/0 (socket says 172.21.15.78:36718) 2024-03-16T11:08:09.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b118f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ac101810 0x7f66ac075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b098e640 1 -- 172.21.15.78:0/586051071 learned_addr learned my addr 172.21.15.78:0/586051071 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac10d320 con 0x7f66ac101810 2024-03-16T11:08:09.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac10d1a0 con 0x7f66ac102a00 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac10d4a0 con 0x7f66ac1033c0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.610+0000 7f66b1990640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66ac1033c0 0x7f66ac074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.611+0000 7f66b1990640 1 -- 172.21.15.78:0/586051071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 msgr2=0x7f66ac073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.611+0000 7f66b1990640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 0x7f66ac073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.611+0000 7f66b1990640 1 -- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ac101810 msgr2=0x7f66ac075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.611+0000 7f66b1990640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ac101810 0x7f66ac075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.611+0000 7f66b1990640 1 -- 172.21.15.78:0/586051071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ac0749a0 con 0x7f66ac1033c0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.611+0000 7f66b1990640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66ac1033c0 0x7f66ac074110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009fd0 tx=0x7f66a400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.612+0000 7f66a27fc640 1 -- 172.21.15.78:0/586051071 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a400cd60 con 0x7f66ac1033c0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.612+0000 7f66a27fc640 1 -- 172.21.15.78:0/586051071 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a4005020 con 0x7f66ac1033c0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.612+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ac1976d0 con 0x7f66ac1033c0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.612+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ac197c20 con 0x7f66ac1033c0 2024-03-16T11:08:09.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.613+0000 7f66a27fc640 1 -- 172.21.15.78:0/586051071 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4010670 con 0x7f66ac1033c0 2024-03-16T11:08:09.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.615+0000 7f66a27fc640 1 -- 172.21.15.78:0/586051071 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f66a40051d0 con 0x7f66ac1033c0 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 Processor -- start 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 -- start start 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c0425c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c042ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c042b3b0 0x55b8c042d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b8c038a490 con 0x55b8c042b3b0 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b8c0332c90 con 0x55b8c0425850 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66b5fdb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b8c0386530 con 0x55b8c0426240 2024-03-16T11:08:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.616+0000 7f66a27fc640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f66980a2900 0x7f66980a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c042ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b098e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c0425c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c042ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50574/0 (socket says 172.21.15.78:50574) 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 -- 172.21.15.78:0/2345477643 learned_addr learned my addr 172.21.15.78:0/2345477643 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66a27fc640 1 -- 172.21.15.78:0/586051071 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f66a40cf160 con 0x7f66ac1033c0 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b098e640 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c042b3b0 0x55b8c042d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b1990640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f66980a2900 0x7f66980a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 -- 172.21.15.78:0/2345477643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 msgr2=0x55b8c0425c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c0425c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 -- 172.21.15.78:0/2345477643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c042b3b0 msgr2=0x55b8c042d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c042b3b0 0x55b8c042d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 -- 172.21.15.78:0/2345477643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8c03ed500 con 0x55b8c0426240 2024-03-16T11:08:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.617+0000 7f66b118f640 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c042ad40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f669c009ca0 tx=0x7f669c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c27e0b16b831c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.618+0000 7f66b1990640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f66980a2900 0x7f66980a4dc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f66a401a040 tx=0x7f66a400c7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.618+0000 7f66977fe640 1 -- 172.21.15.78:0/2345477643 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669c004230 con 0x55b8c0426240 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.618+0000 7f66977fe640 1 -- 172.21.15.78:0/2345477643 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f669c0043d0 con 0x55b8c0426240 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.618+0000 7f66977fe640 1 -- 172.21.15.78:0/2345477643 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669c004cc0 con 0x55b8c0426240 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2345477643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 msgr2=0x55b8c042ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c042ad40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f669c009ca0 tx=0x7f669c011810 comp rx=0 tx=0).stop 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2345477643 shutdown_connections 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c042b3b0 0x55b8c042d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c042ad40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2345477643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c0425c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2345477643 >> 172.21.15.78:0/2345477643 conn(0x55b8c0420dc0 msgr2=0x55b8c04211c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2345477643 shutdown_connections 2024-03-16T11:08:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.619+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2345477643 wait complete. 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.620+0000 7f66b5fdb740 1 -- start start 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.620+0000 7f66b5fdb740 1 Processor -- start 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.620+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0425850 0x55b8c042fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.620+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04302d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.620+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c042b3b0 0x55b8c0437310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b118f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c042b3b0 0x55b8c0437310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0425850 0x55b8c042fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b118f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04302d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36734/0 (socket says 172.21.15.78:36734) 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0425850 0x55b8c042fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41790/0 (socket says 172.21.15.78:41790) 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b118f640 1 -- 172.21.15.78:0/3402062709 learned_addr learned my addr 172.21.15.78:0/3402062709 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b8c042e3b0 con 0x55b8c0425850 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b8c0346230 con 0x55b8c0426240 2024-03-16T11:08:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b8c042e530 con 0x55b8c042b3b0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b5fdb740 10 client.0 _ll_register_callbacks cb 0x55b8c0382010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 -- 172.21.15.78:0/3402062709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 msgr2=0x55b8c04302d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04302d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 -- 172.21.15.78:0/3402062709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0425850 msgr2=0x55b8c042fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0425850 0x55b8c042fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.621+0000 7f66b098e640 1 -- 172.21.15.78:0/3402062709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8c0355100 con 0x55b8c042b3b0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f66b098e640 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c042b3b0 0x55b8c0437310 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66a802c1a0 tx=0x7f66a802bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8909b94c96a40db9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f6694ff9640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f6694ff9640 1 -- 172.21.15.78:0/3402062709 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8056060 con 0x55b8c042b3b0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f669c0043d0 con 0x55b8c042b3b0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f6694ff9640 1 -- 172.21.15.78:0/3402062709 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f66a8004260 con 0x55b8c042b3b0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f6694ff9640 1 -- 172.21.15.78:0/3402062709 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f66a8004460 con 0x55b8c042b3b0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f66b5fdb740 10 client.25828 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.622+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b8c009e3f0 con 0x55b8c042b3b0 2024-03-16T11:08:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f6694ff9640 1 -- 172.21.15.78:0/3402062709 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f669c0043d0 con 0x55b8c042b3b0 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f6694ff9640 1 -- 172.21.15.78:0/3402062709 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f66a805b070 con 0x55b8c042b3b0 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f66b5fdb740 10 client.25828 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f66b5fdb740 10 client.25828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f66b5fdb740 20 client.25828 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c043bb70 0x55b8c043dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f66b5fdb740 4 client.25828 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8c03979d0 con 0x55b8c043bb70 2024-03-16T11:08:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.623+0000 7f66b118f640 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c043bb70 0x55b8c043dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.624+0000 7f66b118f640 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c043bb70 0x55b8c043dfe0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b62346bcb36d95f server_cookie=997a550ff6e62f99 in_seq=0 out_seq=0 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.624+0000 7f6694ff9640 10 client.25828 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f6694ff9640 1 -- 172.21.15.78:0/3402062709 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b8c03979d0 con 0x55b8c043bb70 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f6694ff9640 10 client.25828 handle_command_reply: tid=0 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 client.25828 shutdown 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c042b3b0 msgr2=0x55b8c0437310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c042b3b0 0x55b8c0437310 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66a802c1a0 tx=0x7f66a802bb90 comp rx=0 tx=0).stop 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 shutdown_connections 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c043bb70 0x55b8c043dfe0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c042b3b0 0x55b8c0437310 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04302d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f6694ff9640 0 client.25828 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/3402062709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0425850 0x55b8c042fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 >> 172.21.15.78:0/3402062709 conn(0x55b8c0420dc0 msgr2=0x55b8c041ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.625+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 shutdown_connections 2024-03-16T11:08:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.626+0000 7f66b5fdb740 1 -- 172.21.15.78:0/3402062709 wait complete. 2024-03-16T11:08:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.626+0000 7f66b5fdb740 20 client.25828 trim_cache size 0 max 16384 2024-03-16T11:08:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.648+0000 7f66b5fdb740 1 Processor -- start 2024-03-16T11:08:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.648+0000 7f66b5fdb740 1 -- start start 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0424730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0425850 0x55b8c04372a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04428a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b098e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0424730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b098e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0424730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41798/0 (socket says 172.21.15.78:41798) 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b098e640 1 -- 172.21.15.78:0/155510699 learned_addr learned my addr 172.21.15.78:0/155510699 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1a9310 con 0x55b8c0420dc0 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1a6be0 con 0x55b8c0426240 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1a4a10 con 0x55b8c0425850 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.649+0000 7f66b098e640 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.650+0000 7f66b098e640 1 -- 172.21.15.78:0/155510699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0425850 msgr2=0x55b8c04372a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T11:08:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.650+0000 7f66b098e640 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0425850 0x55b8c04372a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.650+0000 7f66b098e640 1 -- 172.21.15.78:0/155510699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 msgr2=0x55b8c04428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.650+0000 7f66b098e640 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.650+0000 7f66b098e640 1 -- 172.21.15.78:0/155510699 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8c01687d0 con 0x55b8c0420dc0 2024-03-16T11:08:09.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.651+0000 7f66b098e640 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0424730 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009780 tx=0x7f66a8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a98137ee1f0f2d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.651+0000 7f6696ffd640 1 -- 172.21.15.78:0/155510699 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8056070 con 0x55b8c0420dc0 2024-03-16T11:08:09.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.651+0000 7f6696ffd640 1 -- 172.21.15.78:0/155510699 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a8040e30 con 0x55b8c0420dc0 2024-03-16T11:08:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.651+0000 7f6696ffd640 1 -- 172.21.15.78:0/155510699 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8005020 con 0x55b8c0420dc0 2024-03-16T11:08:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 msgr2=0x55b8c0424730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0424730 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009780 tx=0x7f66a8005bd0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 shutdown_connections 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0426240 0x55b8c04428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0425850 0x55b8c04372a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/155510699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0424730 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 >> 172.21.15.78:0/155510699 conn(0x55b8c0427020 msgr2=0x55b8c0424020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.652+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 shutdown_connections 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.653+0000 7f66b5fdb740 1 -- 172.21.15.78:0/155510699 wait complete. 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.653+0000 7f66b5fdb740 1 -- start start 2024-03-16T11:08:09.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.653+0000 7f66b5fdb740 1 Processor -- start 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.653+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0435280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c04357c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b5fdb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c04367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b5fdb740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1a50a0 con 0x55b8c0420dc0 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b5fdb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1a6510 con 0x55b8c0425850 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b5fdb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1a5770 con 0x55b8c0426240 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b5fdb740 10 client.0 _ll_register_callbacks cb 0x7f66a80096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b098e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c04367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b098e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c04367d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50584/0 (socket says 172.21.15.78:50584) 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b098e640 1 -- 172.21.15.78:0/2291503591 learned_addr learned my addr 172.21.15.78:0/2291503591 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b098e640 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0435280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.654+0000 7f66b118f640 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c04357c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.655+0000 7f66b098e640 1 -- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 msgr2=0x55b8c04357c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.655+0000 7f66b098e640 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c04357c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.655+0000 7f66b098e640 1 -- 172.21.15.78:0/2291503591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 msgr2=0x55b8c0435280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.655+0000 7f66b098e640 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0435280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.655+0000 7f66b098e640 1 -- 172.21.15.78:0/2291503591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8c0436dc0 con 0x55b8c0426240 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.655+0000 7f66b098e640 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c04367d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009830 tx=0x7f66a8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f9cb6ef950ee01 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.655+0000 7f6694ff9640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.656+0000 7f6694ff9640 1 -- 172.21.15.78:0/2291503591 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8061040 con 0x55b8c0426240 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.656+0000 7f6694ff9640 1 -- 172.21.15.78:0/2291503591 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f66a8034440 con 0x55b8c0426240 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.656+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a8040e30 con 0x55b8c0426240 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.656+0000 7f6694ff9640 1 -- 172.21.15.78:0/2291503591 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f66a8056040 con 0x55b8c0426240 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.656+0000 7f66b5fdb740 10 client.25831 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.656+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b8c04a26a0 con 0x55b8c0426240 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.656+0000 7f6694ff9640 1 -- 172.21.15.78:0/2291503591 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a8040e30 con 0x55b8c0426240 2024-03-16T11:08:09.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.657+0000 7f6694ff9640 1 -- 172.21.15.78:0/2291503591 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f66a805b040 con 0x55b8c0426240 2024-03-16T11:08:09.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.657+0000 7f66b5fdb740 10 client.25831 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:09.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.657+0000 7f66b5fdb740 10 client.25831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:09.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.657+0000 7f66b5fdb740 20 client.25831 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:09.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.657+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c042b3b0 0x55b8c04a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:09.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.657+0000 7f66b5fdb740 4 client.25831 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:08:09.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.657+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b8c0391d20 con 0x55b8c042b3b0 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.658+0000 7f66b118f640 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c042b3b0 0x55b8c04a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.658+0000 7f66b118f640 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c042b3b0 0x55b8c04a4fb0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4c9f06d986b5561 server_cookie=9ec3b378096af8bb in_seq=0 out_seq=0 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.658+0000 7f6694ff9640 10 client.25831 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.659+0000 7f6694ff9640 1 -- 172.21.15.78:0/2291503591 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b8c0391d20 con 0x55b8c042b3b0 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.659+0000 7f6694ff9640 10 client.25831 handle_command_reply: tid=0 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.659+0000 7f66b5fdb740 1 client.25831 shutdown 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.659+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 msgr2=0x55b8c04367d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.659+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c04367d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009830 tx=0x7f66a8005850 comp rx=0 tx=0).stop 2024-03-16T11:08:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 shutdown_connections 2024-03-16T11:08:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55b8c042b3b0 0x55b8c04a4fb0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b8c0426240 0x55b8c04367d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f6694ff9640 0 client.25831 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b8c0425850 0x55b8c04357c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 --2- 172.21.15.78:0/2291503591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b8c0420dc0 0x55b8c0435280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 >> 172.21.15.78:0/2291503591 conn(0x55b8c0427020 msgr2=0x55b8c04248b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 shutdown_connections 2024-03-16T11:08:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.664+0000 7f66b5fdb740 1 -- 172.21.15.78:0/2291503591 wait complete. 2024-03-16T11:08:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.665+0000 7f66b5fdb740 20 client.25831 trim_cache size 0 max 16384 2024-03-16T11:08:09.667 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.667+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f66980a2900 msgr2=0x7f66980a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f66980a2900 0x7f66980a4dc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f66a401a040 tx=0x7f66a400c7d0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66ac1033c0 msgr2=0x7f66ac074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66ac1033c0 0x7f66ac074110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009fd0 tx=0x7f66a400ee50 comp rx=0 tx=0).stop 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 shutdown_connections 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f66980a2900 0x7f66980a4dc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66ac1033c0 0x7f66ac074110 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ac102a00 0x7f66ac073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 --2- 172.21.15.78:0/586051071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ac101810 0x7f66ac075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 >> 172.21.15.78:0/586051071 conn(0x7f66ac0fcd80 msgr2=0x7f66ac105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.668+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 shutdown_connections 2024-03-16T11:08:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:09.669+0000 7f66b33f3640 1 -- 172.21.15.78:0/586051071 wait complete. 2024-03-16T11:08:10.682 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T11:08:10.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:08:10.682 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 -- 172.21.15.78:0/1133465441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 msgr2=0x7f01e81046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 --2- 172.21.15.78:0/1133465441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e81046b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009ca0 tx=0x7f01e002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 -- 172.21.15.78:0/1133465441 shutdown_connections 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 --2- 172.21.15.78:0/1133465441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e8107520 0x7f01e8109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 --2- 172.21.15.78:0/1133465441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e8104bf0 0x7f01e8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 --2- 172.21.15.78:0/1133465441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e81046b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 -- 172.21.15.78:0/1133465441 >> 172.21.15.78:0/1133465441 conn(0x7f01e80fbc70 msgr2=0x7f01e80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 -- 172.21.15.78:0/1133465441 shutdown_connections 2024-03-16T11:08:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.838+0000 7f01ee987640 1 -- 172.21.15.78:0/1133465441 wait complete. 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.839+0000 7f01ee987640 1 Processor -- start 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.839+0000 7f01ee987640 1 -- start start 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.839+0000 7f01ee987640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e8196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.839+0000 7f01ee987640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e8104bf0 0x7f01e81968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.839+0000 7f01ee987640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e8107520 0x7f01e8192ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01ee987640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01e810d330 con 0x7f01e8107520 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01ee987640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01e810d1b0 con 0x7f01e81022c0 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01ee987640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01e810d4b0 con 0x7f01e8104bf0 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e8196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e8196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36750/0 (socket says 172.21.15.78:36750) 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 -- 172.21.15.78:0/3027223079 learned_addr learned my addr 172.21.15.78:0/3027223079 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e77fe640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e8104bf0 0x7f01e81968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01ecf24640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e8107520 0x7f01e8192ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 -- 172.21.15.78:0/3027223079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e8104bf0 msgr2=0x7f01e81968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e8104bf0 0x7f01e81968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 -- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e8107520 msgr2=0x7f01e8192ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e8107520 0x7f01e8192ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 -- 172.21.15.78:0/3027223079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e81937b0 con 0x7f01e81022c0 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e77fe640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e8104bf0 0x7f01e81968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.840+0000 7f01e7fff640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e8196380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f01e0037040 tx=0x7f01e0032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.841+0000 7f01e57fa640 1 -- 172.21.15.78:0/3027223079 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0039040 con 0x7f01e81022c0 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.841+0000 7f01e57fa640 1 -- 172.21.15.78:0/3027223079 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01e0044070 con 0x7f01e81022c0 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.841+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e81a5f30 con 0x7f01e81022c0 2024-03-16T11:08:10.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.841+0000 7f01e57fa640 1 -- 172.21.15.78:0/3027223079 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0004540 con 0x7f01e81022c0 2024-03-16T11:08:10.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.841+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e81a62e0 con 0x7f01e81022c0 2024-03-16T11:08:10.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.844+0000 7f01e57fa640 1 -- 172.21.15.78:0/3027223079 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f01e00046e0 con 0x7f01e81022c0 2024-03-16T11:08:10.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.844+0000 7f01f1570740 1 Processor -- start 2024-03-16T11:08:10.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.844+0000 7f01f1570740 1 -- start start 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01f1570740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4761db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01f1570740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae4766e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01f1570740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae4769980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01f1570740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dae46c60d0 con 0x55dae4761990 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01f1570740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dae466ec70 con 0x55dae47674f0 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01f1570740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dae46c2170 con 0x55dae4762380 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01ecf24640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4761db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01ecf24640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4761db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41840/0 (socket says 172.21.15.78:41840) 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae4769980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01ecf24640 1 -- 172.21.15.78:0/3221853787 learned_addr learned my addr 172.21.15.78:0/3221853787 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e57fa640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01d00a2880 0x7f01d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e7fff640 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae4766e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e57fa640 1 -- 172.21.15.78:0/3027223079 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f01e00f1200 con 0x7f01e81022c0 2024-03-16T11:08:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01ecf24640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01d00a2880 0x7f01d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e77fe640 1 -- 172.21.15.78:0/3221853787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 msgr2=0x55dae4766e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e77fe640 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae4766e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e77fe640 1 -- 172.21.15.78:0/3221853787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 msgr2=0x55dae4761db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e77fe640 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4761db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.845+0000 7f01e77fe640 1 -- 172.21.15.78:0/3221853787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae44a27e0 con 0x55dae47674f0 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.846+0000 7f01e77fe640 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae4769980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01e8069440 tx=0x7f01dc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a542287c9ff06fa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.846+0000 7f01ecf24640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01d00a2880 0x7f01d00a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55dae4762be0 tx=0x7f01d8004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.846+0000 7f01ce7fc640 1 -- 172.21.15.78:0/3221853787 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc004310 con 0x55dae47674f0 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.846+0000 7f01ce7fc640 1 -- 172.21.15.78:0/3221853787 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01dc0044b0 con 0x55dae47674f0 2024-03-16T11:08:10.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.846+0000 7f01ce7fc640 1 -- 172.21.15.78:0/3221853787 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc0035f0 con 0x55dae47674f0 2024-03-16T11:08:10.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 -- 172.21.15.78:0/3221853787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 msgr2=0x55dae4769980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae4769980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01e8069440 tx=0x7f01dc0118d0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 -- 172.21.15.78:0/3221853787 shutdown_connections 2024-03-16T11:08:10.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae4769980 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae4766e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 --2- 172.21.15.78:0/3221853787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4761db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 -- 172.21.15.78:0/3221853787 >> 172.21.15.78:0/3221853787 conn(0x55dae4729c50 msgr2=0x55dae475d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:10.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 -- 172.21.15.78:0/3221853787 shutdown_connections 2024-03-16T11:08:10.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 -- 172.21.15.78:0/3221853787 wait complete. 2024-03-16T11:08:10.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.848+0000 7f01f1570740 1 -- start start 2024-03-16T11:08:10.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.849+0000 7f01f1570740 1 Processor -- start 2024-03-16T11:08:10.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.849+0000 7f01f1570740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4761990 0x55dae476bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01f1570740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae476c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01f1570740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae47674f0 0x55dae4773210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01f1570740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dae476a4a0 con 0x55dae47674f0 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4761990 0x55dae476bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4761990 0x55dae476bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36754/0 (socket says 172.21.15.78:36754) 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 -- 172.21.15.78:0/1506970457 learned_addr learned my addr 172.21.15.78:0/1506970457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e77fe640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae476c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01ecf24640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae47674f0 0x55dae4773210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dae46820d0 con 0x55dae4761990 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dae476a620 con 0x55dae4762380 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01f1570740 10 client.0 _ll_register_callbacks cb 0x55dae464b760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 -- 172.21.15.78:0/1506970457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 msgr2=0x55dae476c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae476c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 -- 172.21.15.78:0/1506970457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae47674f0 msgr2=0x55dae4773210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:10.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae47674f0 0x55dae4773210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01e7fff640 1 -- 172.21.15.78:0/1506970457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae4502f70 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.850+0000 7f01ecf24640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae47674f0 0x55dae4773210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01e7fff640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4761990 0x55dae476bc90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01e00049d0 tx=0x7f01e00bd720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=defb725d20385cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01c77fe640 1 -- 172.21.15.78:0/1506970457 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e00cf020 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01c77fe640 1 -- 172.21.15.78:0/1506970457 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f01e00eedd0 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01dc0044b0 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01c77fe640 1 -- 172.21.15.78:0/1506970457 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f01e00be260 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01f1570740 10 client.25865 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55dae439a170 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.851+0000 7f01c77fe640 1 -- 172.21.15.78:0/1506970457 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01e00f0030 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01c77fe640 1 -- 172.21.15.78:0/1506970457 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f01e003c030 con 0x55dae4761990 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01f1570740 10 client.25865 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:10.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01f1570740 10 client.25865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:10.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01f1570740 20 client.25865 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:10.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01f1570740 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae4777870 0x55dae4779cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01f1570740 4 client.25865 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:10.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f01dc0044b0 con 0x55dae4777870 2024-03-16T11:08:10.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01e77fe640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae4777870 0x55dae4779cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.853+0000 7f01c77fe640 10 client.25865 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:10.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.852+0000 7f01e77fe640 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae4777870 0x55dae4779cf0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e51e8f06de230d6 server_cookie=d6150ce1a7b0cb6a in_seq=0 out_seq=0 2024-03-16T11:08:10.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.853+0000 7f01c77fe640 1 -- 172.21.15.78:0/1506970457 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f01dc0044b0 con 0x55dae4777870 2024-03-16T11:08:10.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.853+0000 7f01c77fe640 10 client.25865 handle_command_reply: tid=0 2024-03-16T11:08:10.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.853+0000 7f01f1570740 1 client.25865 shutdown 2024-03-16T11:08:10.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4761990 msgr2=0x55dae476bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4761990 0x55dae476bc90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01e00049d0 tx=0x7f01e00bd720 comp rx=0 tx=0).stop 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 shutdown_connections 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae4777870 0x55dae4779cf0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae47674f0 0x55dae4773210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4762380 0x55dae476c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 --2- 172.21.15.78:0/1506970457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4761990 0x55dae476bc90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 >> 172.21.15.78:0/1506970457 conn(0x55dae4729c50 msgr2=0x55dae475c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01c77fe640 0 client.25865 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 shutdown_connections 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 1 -- 172.21.15.78:0/1506970457 wait complete. 2024-03-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.854+0000 7f01f1570740 20 client.25865 trim_cache size 0 max 16384 2024-03-16T11:08:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.882+0000 7f01f1570740 1 Processor -- start 2024-03-16T11:08:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.882+0000 7f01f1570740 1 -- start start 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01f1570740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4761990 0x55dae4760c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01f1570740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4762380 0x55dae47731d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01f1570740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae476b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01f1570740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01e81b19c0 con 0x55dae4762380 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01f1570740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01e81af290 con 0x55dae47674f0 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01f1570740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01e81ad0c0 con 0x55dae4761990 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae476b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae476b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36770/0 (socket says 172.21.15.78:36770) 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 -- 172.21.15.78:0/2504501813 learned_addr learned my addr 172.21.15.78:0/2504501813 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01e7fff640 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4761990 0x55dae4760c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 -- 172.21.15.78:0/2504501813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4761990 msgr2=0x55dae4760c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4761990 0x55dae4760c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 -- 172.21.15.78:0/2504501813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4762380 msgr2=0x55dae47731d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4762380 0x55dae47731d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.883+0000 7f01ecf24640 1 -- 172.21.15.78:0/2504501813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae4763d20 con 0x55dae47674f0 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.884+0000 7f01ecf24640 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae476b8a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f01d80042f0 tx=0x7f01d8009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3586c927722152f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.884+0000 7f01cdffb640 1 -- 172.21.15.78:0/2504501813 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8012560 con 0x55dae47674f0 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.884+0000 7f01cdffb640 1 -- 172.21.15.78:0/2504501813 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01d8012bb0 con 0x55dae47674f0 2024-03-16T11:08:10.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.884+0000 7f01cdffb640 1 -- 172.21.15.78:0/2504501813 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8022b30 con 0x55dae47674f0 2024-03-16T11:08:10.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.885+0000 7f01f1570740 1 -- 172.21.15.78:0/2504501813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 msgr2=0x55dae476b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.885+0000 7f01f1570740 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae476b8a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f01d80042f0 tx=0x7f01d8009880 comp rx=0 tx=0).stop 2024-03-16T11:08:10.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.885+0000 7f01f1570740 1 -- 172.21.15.78:0/2504501813 shutdown_connections 2024-03-16T11:08:10.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.885+0000 7f01f1570740 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae47674f0 0x55dae476b8a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.885+0000 7f01f1570740 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4762380 0x55dae47731d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.885+0000 7f01f1570740 1 --2- 172.21.15.78:0/2504501813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae4761990 0x55dae4760c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.885+0000 7f01f1570740 1 -- 172.21.15.78:0/2504501813 >> 172.21.15.78:0/2504501813 conn(0x55dae4729c50 msgr2=0x55dae4760540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:10.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.886+0000 7f01f1570740 1 -- 172.21.15.78:0/2504501813 shutdown_connections 2024-03-16T11:08:10.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.886+0000 7f01f1570740 1 -- 172.21.15.78:0/2504501813 wait complete. 2024-03-16T11:08:10.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.886+0000 7f01f1570740 1 -- start start 2024-03-16T11:08:10.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.886+0000 7f01f1570740 1 Processor -- start 2024-03-16T11:08:10.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.886+0000 7f01f1570740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4772a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.886+0000 7f01f1570740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4762380 0x55dae4772fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01f1570740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae47674f0 0x55dae47710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01f1570740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01e81ad750 con 0x55dae4761990 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01f1570740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01e81aebc0 con 0x55dae4762380 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01f1570740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01e81ade20 con 0x55dae47674f0 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4762380 0x55dae4772fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4762380 0x55dae4772fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36778/0 (socket says 172.21.15.78:36778) 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01f1570740 10 client.0 _ll_register_callbacks cb 0x7f01e00bda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01e77fe640 1 -- 172.21.15.78:0/1656644797 learned_addr learned my addr 172.21.15.78:0/1656644797 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01ecf24640 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae47674f0 0x55dae47710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01ecf24640 1 -- 172.21.15.78:0/1656644797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4762380 msgr2=0x55dae4772fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01ecf24640 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4762380 0x55dae4772fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01ecf24640 1 -- 172.21.15.78:0/1656644797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 msgr2=0x55dae4772a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01ecf24640 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4772a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.887+0000 7f01ecf24640 1 -- 172.21.15.78:0/1656644797 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae4771790 con 0x55dae47674f0 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01ecf24640 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae47674f0 0x55dae47710a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f01d800ff80 tx=0x7f01d8009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bdb4db5862a1e41 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01c77fe640 1 -- 172.21.15.78:0/1656644797 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d80228b0 con 0x55dae47674f0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d8012bb0 con 0x55dae47674f0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01c77fe640 1 -- 172.21.15.78:0/1656644797 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f01d8022a50 con 0x55dae47674f0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01c77fe640 1 -- 172.21.15.78:0/1656644797 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f01d803b050 con 0x55dae47674f0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01f1570740 10 client.25849 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55dae47ef640 con 0x55dae47674f0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.888+0000 7f01c77fe640 1 -- 172.21.15.78:0/1656644797 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01d8012bb0 con 0x55dae47674f0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01c77fe640 1 -- 172.21.15.78:0/1656644797 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f01d801f040 con 0x55dae47674f0 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01f1570740 10 client.25849 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01f1570740 10 client.25849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01f1570740 20 client.25849 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01f1570740 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae47eff80 0x55dae47f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01f1570740 4 client.25849 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dae46d3610 con 0x55dae47eff80 2024-03-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.889+0000 7f01e7fff640 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae47eff80 0x55dae47f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.890+0000 7f01c77fe640 10 client.25849 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.890+0000 7f01e7fff640 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae47eff80 0x55dae47f23a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38fb2fe2821d7cb8 server_cookie=976720a88988110d in_seq=0 out_seq=0 2024-03-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01c77fe640 1 -- 172.21.15.78:0/1656644797 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dae46d3610 con 0x55dae47eff80 2024-03-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01c77fe640 10 client.25849 handle_command_reply: tid=0 2024-03-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 client.25849 shutdown 2024-03-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae47674f0 msgr2=0x55dae47710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae47674f0 0x55dae47710a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f01d800ff80 tx=0x7f01d8009400 comp rx=0 tx=0).stop 2024-03-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 shutdown_connections 2024-03-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55dae47eff80 0x55dae47f23a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae47674f0 0x55dae47710a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dae4762380 0x55dae4772fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 --2- 172.21.15.78:0/1656644797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dae4761990 0x55dae4772a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 >> 172.21.15.78:0/1656644797 conn(0x55dae4729c50 msgr2=0x55dae476b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01c77fe640 0 client.25849 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 shutdown_connections 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.893+0000 7f01f1570740 1 -- 172.21.15.78:0/1656644797 wait complete. 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.894+0000 7f01f1570740 20 client.25849 trim_cache size 0 max 16384 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "a97eaa8a-5555-471f-b820-95c527cd06fb", 2024-03-16T11:08:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-16T11:08:10.896 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.894+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01d00a2880 msgr2=0x7f01d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.894+0000 7f01ee987640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01d00a2880 0x7f01d00a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55dae4762be0 tx=0x7f01d8004730 comp rx=0 tx=0).stop 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.894+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 msgr2=0x7f01e8196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.894+0000 7f01ee987640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e8196380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f01e0037040 tx=0x7f01e0032340 comp rx=0 tx=0).stop 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 shutdown_connections 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e8107520 0x7f01e8192ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01d00a2880 0x7f01d00a4d40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e8104bf0 0x7f01e81968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 --2- 172.21.15.78:0/3027223079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e81022c0 0x7f01e8196380 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 >> 172.21.15.78:0/3027223079 conn(0x7f01e80fbc70 msgr2=0x7f01e81019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 shutdown_connections 2024-03-16T11:08:10.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:10.895+0000 7f01ee987640 1 -- 172.21.15.78:0/3027223079 wait complete. 2024-03-16T11:08:10.912 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:08:10.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:08:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.073+0000 7f59435d4640 1 -- 172.21.15.78:0/148895384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 msgr2=0x7f593c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.073+0000 7f59435d4640 1 --2- 172.21.15.78:0/148895384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 0x7f593c108b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5934009ca0 tx=0x7f593402c100 comp rx=0 tx=0).stop 2024-03-16T11:08:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 -- 172.21.15.78:0/148895384 shutdown_connections 2024-03-16T11:08:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 --2- 172.21.15.78:0/148895384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 0x7f593c108b20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 --2- 172.21.15.78:0/148895384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1018d0 0x7f593c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 --2- 172.21.15.78:0/148895384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f593c1006e0 0x7f593c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 -- 172.21.15.78:0/148895384 >> 172.21.15.78:0/148895384 conn(0x7f593c0fbc70 msgr2=0x7f593c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 -- 172.21.15.78:0/148895384 shutdown_connections 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 -- 172.21.15.78:0/148895384 wait complete. 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.074+0000 7f59435d4640 1 Processor -- start 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f59435d4640 1 -- start start 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f59435d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1006e0 0x7f593c1957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f59435d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f593c1018d0 0x7f593c195ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f59435d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 0x7f593c19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f5941370640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1006e0 0x7f593c1957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f5941370640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1006e0 0x7f593c1957a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36800/0 (socket says 172.21.15.78:36800) 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f5941370640 1 -- 172.21.15.78:0/143970016 learned_addr learned my addr 172.21.15.78:0/143970016 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10b110 con 0x7f593c1018d0 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10af90 con 0x7f593c1006e0 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.075+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10b290 con 0x7f593c102290 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5940b6f640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f593c1018d0 0x7f593c195ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5941b71640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 0x7f593c19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5941370640 1 -- 172.21.15.78:0/143970016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 msgr2=0x7f593c19cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5941370640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 0x7f593c19cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5941370640 1 -- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f593c1018d0 msgr2=0x7f593c195ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5941370640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f593c1018d0 0x7f593c195ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5941370640 1 -- 172.21.15.78:0/143970016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c19d3c0 con 0x7f593c1006e0 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5940b6f640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f593c1018d0 0x7f593c195ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f5941370640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1006e0 0x7f593c1957a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5938009980 tx=0x7f5938009e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.076+0000 7f59327fc640 1 -- 172.21.15.78:0/143970016 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5938007a70 con 0x7f593c1006e0 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.077+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593c19d6b0 con 0x7f593c1006e0 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.077+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f593c19dc50 con 0x7f593c1006e0 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.077+0000 7f59327fc640 1 -- 172.21.15.78:0/143970016 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5938007c10 con 0x7f593c1006e0 2024-03-16T11:08:11.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.077+0000 7f59327fc640 1 -- 172.21.15.78:0/143970016 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5938003d80 con 0x7f593c1006e0 2024-03-16T11:08:11.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.080+0000 7f59461c1740 1 Processor -- start 2024-03-16T11:08:11.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.080+0000 7f59461c1740 1 -- start start 2024-03-16T11:08:11.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.080+0000 7f59327fc640 1 -- 172.21.15.78:0/143970016 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f593801e030 con 0x7f593c1006e0 2024-03-16T11:08:11.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.080+0000 7f59461c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f576e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.080+0000 7f59461c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576ef70 0x5598f5773a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.080+0000 7f59461c1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f57740e0 0x5598f5776570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f59461c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5598f56cf500 con 0x5598f576e580 2024-03-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f59461c1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5598f5677be0 con 0x5598f57740e0 2024-03-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f59461c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5598f56cb5a0 con 0x5598f576ef70 2024-03-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f5941370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576ef70 0x5598f5773a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f5941370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576ef70 0x5598f5773a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50644/0 (socket says 172.21.15.78:50644) 2024-03-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f5941370640 1 -- 172.21.15.78:0/3208000901 learned_addr learned my addr 172.21.15.78:0/3208000901 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f5941b71640 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f576e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f5941370640 1 -- 172.21.15.78:0/3208000901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f57740e0 msgr2=0x5598f5776570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.081+0000 7f5941370640 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f57740e0 0x5598f5776570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f5941370640 1 -- 172.21.15.78:0/3208000901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 msgr2=0x5598f576e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f5941370640 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f576e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f5941370640 1 -- 172.21.15.78:0/3208000901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598f568b4b0 con 0x5598f576ef70 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f59327fc640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59280a2880 0x7f59280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f59327fc640 1 -- 172.21.15.78:0/143970016 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f593800c080 con 0x7f593c1006e0 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f5941370640 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576ef70 0x5598f5773a70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f59380d1c00 tx=0x7f59380d66c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6089b281de278933 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f5941b71640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59280a2880 0x7f59280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.082+0000 7f59277fe640 1 -- 172.21.15.78:0/3208000901 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593809c6b0 con 0x5598f576ef70 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.083+0000 7f59277fe640 1 -- 172.21.15.78:0/3208000901 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59380035f0 con 0x5598f576ef70 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.083+0000 7f59277fe640 1 -- 172.21.15.78:0/3208000901 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593809c240 con 0x5598f576ef70 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.083+0000 7f59461c1740 1 -- 172.21.15.78:0/3208000901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576ef70 msgr2=0x5598f5773a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.083+0000 7f59461c1740 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576ef70 0x5598f5773a70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f59380d1c00 tx=0x7f59380d66c0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.084+0000 7f5941b71640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59280a2880 0x7f59280a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5934032380 tx=0x7f593402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.084+0000 7f59461c1740 1 -- 172.21.15.78:0/3208000901 shutdown_connections 2024-03-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.084+0000 7f59461c1740 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f57740e0 0x5598f5776570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.084+0000 7f59461c1740 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576ef70 0x5598f5773a70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.084+0000 7f59461c1740 1 --2- 172.21.15.78:0/3208000901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f576e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.084+0000 7f59461c1740 1 -- 172.21.15.78:0/3208000901 >> 172.21.15.78:0/3208000901 conn(0x5598f57656e0 msgr2=0x5598f5765ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.084+0000 7f59461c1740 1 -- 172.21.15.78:0/3208000901 shutdown_connections 2024-03-16T11:08:11.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.085+0000 7f59461c1740 1 -- 172.21.15.78:0/3208000901 wait complete. 2024-03-16T11:08:11.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 -- start start 2024-03-16T11:08:11.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 Processor -- start 2024-03-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f5778b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57790d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f57740e0 0x5598f5780110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5598f5777090 con 0x5598f576e580 2024-03-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5598f568b160 con 0x5598f576ef70 2024-03-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5598f5777210 con 0x5598f57740e0 2024-03-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.086+0000 7f59461c1740 10 client.0 _ll_register_callbacks cb 0x5598f566f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57790d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57790d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36802/0 (socket says 172.21.15.78:36802) 2024-03-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 -- 172.21.15.78:0/242289072 learned_addr learned my addr 172.21.15.78:0/242289072 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5941b71640 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f57740e0 0x5598f5780110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5941370640 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f5778b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 -- 172.21.15.78:0/242289072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f57740e0 msgr2=0x5598f5780110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f57740e0 0x5598f5780110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 -- 172.21.15.78:0/242289072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 msgr2=0x5598f5778b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f5778b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 -- 172.21.15.78:0/242289072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598f54ad7d0 con 0x5598f576ef70 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5940b6f640 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57790d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f592c011690 tx=0x7f592c011b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70e376dd079570a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.087+0000 7f5924ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.088+0000 7f5924ff9640 1 -- 172.21.15.78:0/242289072 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c01f060 con 0x5598f576ef70 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.088+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5598f56dca40 con 0x5598f576ef70 2024-03-16T11:08:11.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.088+0000 7f5924ff9640 1 -- 172.21.15.78:0/242289072 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f592c00fb40 con 0x5598f576ef70 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.088+0000 7f5924ff9640 1 -- 172.21.15.78:0/242289072 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f592c011e90 con 0x5598f576ef70 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.088+0000 7f59461c1740 10 client.25880 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.088+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5598f53e33f0 con 0x5598f576ef70 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.088+0000 7f5924ff9640 1 -- 172.21.15.78:0/242289072 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5598f56dca40 con 0x5598f576ef70 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f5924ff9640 1 -- 172.21.15.78:0/242289072 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f592c024070 con 0x5598f576ef70 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f59461c1740 10 client.25880 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f59461c1740 10 client.25880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f59461c1740 20 client.25880 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f59461c1740 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f5784a40 0x5598f5786ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f59461c1740 4 client.25880 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598f56d6d90 con 0x5598f5784a40 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f5941370640 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f5784a40 0x5598f5786ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f5941370640 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f5784a40 0x5598f5786ee0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1d3299dd5585335 server_cookie=2db8dc30501d29b8 in_seq=0 out_seq=0 2024-03-16T11:08:11.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.089+0000 7f5924ff9640 10 client.25880 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.090+0000 7f5924ff9640 1 -- 172.21.15.78:0/242289072 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5598f56d6d90 con 0x5598f5784a40 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.090+0000 7f5924ff9640 10 client.25880 handle_command_reply: tid=0 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 client.25880 shutdown 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 msgr2=0x5598f57790d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57790d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f592c011690 tx=0x7f592c011b50 comp rx=0 tx=0).stop 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 shutdown_connections 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f5784a40 0x5598f5786ee0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f57740e0 0x5598f5780110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57790d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f5924ff9640 0 client.25880 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 --2- 172.21.15.78:0/242289072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f576e580 0x5598f5778b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 >> 172.21.15.78:0/242289072 conn(0x5598f57656e0 msgr2=0x5598f5764900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 shutdown_connections 2024-03-16T11:08:11.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.091+0000 7f59461c1740 1 -- 172.21.15.78:0/242289072 wait complete. 2024-03-16T11:08:11.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.092+0000 7f59461c1740 20 client.25880 trim_cache size 0 max 16384 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 Processor -- start 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 -- start start 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f57608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57800d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f5764070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f593c1a9390 con 0x5598f57740e0 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f5941370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f57608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f5941370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f57608e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50658/0 (socket says 172.21.15.78:50658) 2024-03-16T11:08:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f5941370640 1 -- 172.21.15.78:0/3199807362 learned_addr learned my addr 172.21.15.78:0/3199807362 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 -- 172.21.15.78:0/3199807362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f593c1a6c60 con 0x5598f576ef70 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f59461c1740 1 -- 172.21.15.78:0/3199807362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f593c1a4a90 con 0x5598f576e580 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f5941b71640 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f5764070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.119+0000 7f5940b6f640 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57800d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5941b71640 1 -- 172.21.15.78:0/3199807362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 msgr2=0x5598f57608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5941b71640 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f57608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5941b71640 1 -- 172.21.15.78:0/3199807362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 msgr2=0x5598f57800d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5941b71640 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57800d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5941b71640 1 -- 172.21.15.78:0/3199807362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598f53a30f0 con 0x5598f57740e0 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5940b6f640 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57800d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5941370640 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f57608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5941b71640 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f5764070 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f593402c710 tx=0x7f5934040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfdd10843583289 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5926ffd640 1 -- 172.21.15.78:0/3199807362 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934039040 con 0x5598f57740e0 2024-03-16T11:08:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5926ffd640 1 -- 172.21.15.78:0/3199807362 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5934053070 con 0x5598f57740e0 2024-03-16T11:08:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.120+0000 7f5926ffd640 1 -- 172.21.15.78:0/3199807362 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59340406c0 con 0x5598f57740e0 2024-03-16T11:08:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.121+0000 7f59461c1740 1 -- 172.21.15.78:0/3199807362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 msgr2=0x5598f5764070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.121+0000 7f59461c1740 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f5764070 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f593402c710 tx=0x7f5934040000 comp rx=0 tx=0).stop 2024-03-16T11:08:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.121+0000 7f59461c1740 1 -- 172.21.15.78:0/3199807362 shutdown_connections 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.121+0000 7f59461c1740 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f5764070 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.121+0000 7f59461c1740 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f57800d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.121+0000 7f59461c1740 1 --2- 172.21.15.78:0/3199807362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f57608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.121+0000 7f59461c1740 1 -- 172.21.15.78:0/3199807362 >> 172.21.15.78:0/3199807362 conn(0x5598f57656e0 msgr2=0x5598f5770550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.122+0000 7f59461c1740 1 -- 172.21.15.78:0/3199807362 shutdown_connections 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.122+0000 7f59461c1740 1 -- 172.21.15.78:0/3199807362 wait complete. 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.122+0000 7f59461c1740 1 -- start start 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.122+0000 7f59461c1740 1 Processor -- start 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f59461c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f577c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f59461c1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f577e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f59461c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f577eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f59461c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f593c1a5120 con 0x5598f57740e0 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f59461c1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f593c1a6590 con 0x5598f576ef70 2024-03-16T11:08:11.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f59461c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f593c1a57f0 con 0x5598f576e580 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f59461c1740 10 client.0 _ll_register_callbacks cb 0x7f592c009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f5940b6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f577e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f5941370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f577c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f5940b6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f577e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36818/0 (socket says 172.21.15.78:36818) 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f5941370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f577c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50668/0 (socket says 172.21.15.78:50668) 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.123+0000 7f5940b6f640 1 -- 172.21.15.78:0/461362548 learned_addr learned my addr 172.21.15.78:0/461362548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941b71640 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f577eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941370640 1 -- 172.21.15.78:0/461362548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 msgr2=0x5598f577e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941370640 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f577e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941370640 1 -- 172.21.15.78:0/461362548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 msgr2=0x5598f577eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941370640 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f577eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941370640 1 -- 172.21.15.78:0/461362548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598f56969d0 con 0x5598f576e580 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941b71640 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f577eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:11.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5941370640 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f577c0b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f593809c5e0 tx=0x7f5938000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2225e585439f71e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5924ff9640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5924ff9640 1 -- 172.21.15.78:0/461362548 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593809c420 con 0x5598f576e580 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5934053070 con 0x5598f576e580 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5924ff9640 1 -- 172.21.15.78:0/461362548 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f593809a980 con 0x5598f576e580 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f5924ff9640 1 -- 172.21.15.78:0/461362548 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f593809c6b0 con 0x5598f576e580 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f59461c1740 10 client.25867 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.124+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5598f57ef610 con 0x5598f576e580 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f5924ff9640 1 -- 172.21.15.78:0/461362548 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5934053070 con 0x5598f576e580 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f5924ff9640 1 -- 172.21.15.78:0/461362548 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f59380d4070 con 0x5598f576e580 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f59461c1740 10 client.25867 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f59461c1740 10 client.25867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f59461c1740 20 client.25867 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f59461c1740 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f57ef930 0x5598f57f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f59461c1740 4 client.25867 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:08:11.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5598f5777c90 con 0x5598f57ef930 2024-03-16T11:08:11.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.125+0000 7f5940b6f640 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f57ef930 0x5598f57f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f5924ff9640 10 client.25867 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f5940b6f640 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f57ef930 0x5598f57f1d50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b1aa89d59fea9aa server_cookie=8f92a91f0cf21ec6 in_seq=0 out_seq=0 2024-03-16T11:08:11.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f5924ff9640 1 -- 172.21.15.78:0/461362548 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5598f5777c90 con 0x5598f57ef930 2024-03-16T11:08:11.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f5924ff9640 10 client.25867 handle_command_reply: tid=0 2024-03-16T11:08:11.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f59461c1740 1 client.25867 shutdown 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 msgr2=0x5598f577c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f59461c1740 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f577c0b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f593809c5e0 tx=0x7f5938000bc0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 shutdown_connections 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f59461c1740 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5598f57ef930 0x5598f57f1d50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f59461c1740 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5598f57740e0 0x5598f577eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.126+0000 7f59461c1740 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598f576ef70 0x5598f577e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.127+0000 7f59461c1740 1 --2- 172.21.15.78:0/461362548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598f576e580 0x5598f577c0b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.127+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 >> 172.21.15.78:0/461362548 conn(0x5598f57656e0 msgr2=0x5598f5768940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.127+0000 7f5924ff9640 0 client.25867 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.127+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 shutdown_connections 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.127+0000 7f59461c1740 1 -- 172.21.15.78:0/461362548 wait complete. 2024-03-16T11:08:11.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.127+0000 7f59461c1740 20 client.25867 trim_cache size 0 max 16384 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: "a97eaa8a-5555-471f-b820-95c527cd06fb": { 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "a97eaa8a-5555-471f-b820-95c527cd06fb", 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:11.132 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:11.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59280a2880 msgr2=0x7f59280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59280a2880 0x7f59280a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5934032380 tx=0x7f593402e040 comp rx=0 tx=0).stop 2024-03-16T11:08:11.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1006e0 msgr2=0x7f593c1957a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1006e0 0x7f593c1957a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5938009980 tx=0x7f5938009e40 comp rx=0 tx=0).stop 2024-03-16T11:08:11.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 shutdown_connections 2024-03-16T11:08:11.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f59280a2880 0x7f59280a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c102290 0x7f593c19cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f593c1018d0 0x7f593c195ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 --2- 172.21.15.78:0/143970016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1006e0 0x7f593c1957a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 >> 172.21.15.78:0/143970016 conn(0x7f593c0fbc70 msgr2=0x7f593c104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 shutdown_connections 2024-03-16T11:08:11.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:11.128+0000 7f59435d4640 1 -- 172.21.15.78:0/143970016 wait complete. 2024-03-16T11:08:11.144 INFO:tasks.cephfs.filesystem:scrub status for tag:a97eaa8a-5555-471f-b820-95c527cd06fb - {'path': '/', 'tag': 'a97eaa8a-5555-471f-b820-95c527cd06fb', 'options': 'recursive,force'} 2024-03-16T11:08:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:11 smithi098 ceph-mon[35848]: pgmap v414: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 4.25k op/s 2024-03-16T11:08:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:11 smithi098 ceph-mon[35848]: scrub queued for path: / 2024-03-16T11:08:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:11 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:08:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:11 smithi098 ceph-mon[35848]: scrub summary: active paths [/] 2024-03-16T11:08:11.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:11 smithi098 ceph-mon[35848]: scrub summary: idle+waiting paths [/] 2024-03-16T11:08:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:11 smithi153 ceph-mon[35521]: pgmap v414: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 4.25k op/s 2024-03-16T11:08:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:11 smithi153 ceph-mon[35521]: scrub queued for path: / 2024-03-16T11:08:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:11 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:08:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:11 smithi153 ceph-mon[35521]: scrub summary: active paths [/] 2024-03-16T11:08:11.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:11 smithi153 ceph-mon[35521]: scrub summary: idle+waiting paths [/] 2024-03-16T11:08:11.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:11 smithi078 ceph-mon[32235]: pgmap v414: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 19 MiB/s wr, 4.25k op/s 2024-03-16T11:08:11.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:11 smithi078 ceph-mon[32235]: scrub queued for path: / 2024-03-16T11:08:11.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:11 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:08:11.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:11 smithi078 ceph-mon[32235]: scrub summary: active paths [/] 2024-03-16T11:08:11.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:11 smithi078 ceph-mon[32235]: scrub summary: idle+waiting paths [/] 2024-03-16T11:08:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.683+0000 7f75b21d3640 1 -- 172.21.15.78:0/2364894867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac100700 msgr2=0x7f75ac100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.683+0000 7f75b21d3640 1 --2- 172.21.15.78:0/2364894867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac100700 0x7f75ac100b20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f759c009900 tx=0x7f759c02bc40 comp rx=0 tx=0).stop 2024-03-16T11:08:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.683+0000 7f75b21d3640 1 -- 172.21.15.78:0/2364894867 shutdown_connections 2024-03-16T11:08:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.684+0000 7f75b21d3640 1 --2- 172.21.15.78:0/2364894867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75ac1022b0 0x7f75ac108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.684+0000 7f75b21d3640 1 --2- 172.21.15.78:0/2364894867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac1018f0 0x7f75ac101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.684+0000 7f75b21d3640 1 --2- 172.21.15.78:0/2364894867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac100700 0x7f75ac100b20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.684+0000 7f75b21d3640 1 -- 172.21.15.78:0/2364894867 >> 172.21.15.78:0/2364894867 conn(0x7f75ac0fbc70 msgr2=0x7f75ac0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.684+0000 7f75b21d3640 1 -- 172.21.15.78:0/2364894867 shutdown_connections 2024-03-16T11:08:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.684+0000 7f75b21d3640 1 -- 172.21.15.78:0/2364894867 wait complete. 2024-03-16T11:08:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.685+0000 7f75b21d3640 1 Processor -- start 2024-03-16T11:08:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75b21d3640 1 -- start start 2024-03-16T11:08:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75b21d3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac100700 0x7f75ac195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75b21d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75ac1018f0 0x7f75ac195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75b21d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac1022b0 0x7f75ac19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75b21d3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac10b0f0 con 0x7f75ac1022b0 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75b21d3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac10af70 con 0x7f75ac1018f0 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75b21d3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac10b270 con 0x7f75ac100700 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75ac1018f0 0x7f75ac195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75abfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac1022b0 0x7f75ac19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75ab7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac100700 0x7f75ac195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75abfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac1022b0 0x7f75ac19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41922/0 (socket says 172.21.15.78:41922) 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.686+0000 7f75abfff640 1 -- 172.21.15.78:0/3061677344 learned_addr learned my addr 172.21.15.78:0/3061677344 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75aaffd640 1 -- 172.21.15.78:0/3061677344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac100700 msgr2=0x7f75ac195780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75aaffd640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac100700 0x7f75ac195780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75aaffd640 1 -- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac1022b0 msgr2=0x7f75ac19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75aaffd640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac1022b0 0x7f75ac19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75aaffd640 1 -- 172.21.15.78:0/3061677344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac19d3a0 con 0x7f75ac1018f0 2024-03-16T11:08:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac100700 0x7f75ac195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75aaffd640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75ac1018f0 0x7f75ac195cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f75a000d6d0 tx=0x7f75a000db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75a8ff9640 1 -- 172.21.15.78:0/3061677344 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0017070 con 0x7f75ac1018f0 2024-03-16T11:08:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75a8ff9640 1 -- 172.21.15.78:0/3061677344 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75a0009820 con 0x7f75ac1018f0 2024-03-16T11:08:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75a8ff9640 1 -- 172.21.15.78:0/3061677344 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0004c40 con 0x7f75ac1018f0 2024-03-16T11:08:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.687+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ac19d690 con 0x7f75ac1018f0 2024-03-16T11:08:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.688+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ac19db90 con 0x7f75ac1018f0 2024-03-16T11:08:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.688+0000 7f75abfff640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac1022b0 0x7f75ac19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 Processor -- start 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 -- start start 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75a8ff9640 1 -- 172.21.15.78:0/3061677344 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f75a0010460 con 0x7f75ac1018f0 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63a340 0x556e3b63a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63ad30 0x556e3b63f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b642330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556e3b59fa40 con 0x556e3b63ad30 2024-03-16T11:08:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556e3b548920 con 0x556e3b63a340 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75b4db6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556e3b59bae0 con 0x556e3b63fea0 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63a340 0x556e3b63a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63a340 0x556e3b63a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36832/0 (socket says 172.21.15.78:36832) 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75abfff640 1 -- 172.21.15.78:0/2916811255 learned_addr learned my addr 172.21.15.78:0/2916811255 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.691+0000 7f75aaffd640 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b642330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63ad30 0x556e3b63f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75abfff640 1 -- 172.21.15.78:0/2916811255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 msgr2=0x556e3b642330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75abfff640 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b642330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75abfff640 1 -- 172.21.15.78:0/2916811255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63ad30 msgr2=0x556e3b63f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75abfff640 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63ad30 0x556e3b63f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75abfff640 1 -- 172.21.15.78:0/2916811255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e3b560460 con 0x556e3b63a340 2024-03-16T11:08:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75a8ff9640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75940a2930 0x7f75940a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75abfff640 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63a340 0x556e3b63a760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f75a40118b0 tx=0x7f75a4011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a3367b2e11bdcf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75940a2930 0x7f75940a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f75a8ff9640 1 -- 172.21.15.78:0/3061677344 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f75a00ce240 con 0x7f75ac1018f0 2024-03-16T11:08:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f7591ffb640 1 -- 172.21.15.78:0/2916811255 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a401e070 con 0x556e3b63a340 2024-03-16T11:08:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f7591ffb640 1 -- 172.21.15.78:0/2916811255 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75a400fe70 con 0x556e3b63a340 2024-03-16T11:08:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.692+0000 7f7591ffb640 1 -- 172.21.15.78:0/2916811255 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4019d90 con 0x556e3b63a340 2024-03-16T11:08:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 -- 172.21.15.78:0/2916811255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63a340 msgr2=0x556e3b63a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63a340 0x556e3b63a760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f75a40118b0 tx=0x7f75a4011d70 comp rx=0 tx=0).stop 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75940a2930 0x7f75940a4df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f759c005ab0 tx=0x7f759c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 -- 172.21.15.78:0/2916811255 shutdown_connections 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b642330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63ad30 0x556e3b63f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 --2- 172.21.15.78:0/2916811255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63a340 0x556e3b63a760 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 -- 172.21.15.78:0/2916811255 >> 172.21.15.78:0/2916811255 conn(0x556e3b6358b0 msgr2=0x556e3b635cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.693+0000 7f75b4db6740 1 -- 172.21.15.78:0/2916811255 shutdown_connections 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.694+0000 7f75b4db6740 1 -- 172.21.15.78:0/2916811255 wait complete. 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.694+0000 7f75b4db6740 1 -- start start 2024-03-16T11:08:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.694+0000 7f75b4db6740 1 Processor -- start 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 0x556e3b6445f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b644b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b64bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75b4db6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x556e3b642e50 con 0x556e3b63fea0 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b644b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b644b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36840/0 (socket says 172.21.15.78:36840) 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75aaffd640 1 -- 172.21.15.78:0/1824412472 learned_addr learned my addr 172.21.15.78:0/1824412472 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75abfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b64bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556e3b55bd70 con 0x556e3b63ad30 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556e3b642fd0 con 0x556e3b63a340 2024-03-16T11:08:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75b4db6740 10 client.0 _ll_register_callbacks cb 0x556e3b2b3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.695+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 0x556e3b6445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f75aaffd640 1 -- 172.21.15.78:0/1824412472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 msgr2=0x556e3b6445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f75aaffd640 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 0x556e3b6445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f75aaffd640 1 -- 172.21.15.78:0/1824412472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 msgr2=0x556e3b64bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f75aaffd640 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b64bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f75aaffd640 1 -- 172.21.15.78:0/1824412472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e3b2b4210 con 0x556e3b63ad30 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f75aaffd640 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b644b30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f75a00970f0 tx=0x7f75a0096ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb838e5f05e491fe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f758affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f758affd640 1 -- 172.21.15.78:0/1824412472 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0020040 con 0x556e3b63ad30 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a400fe70 con 0x556e3b63ad30 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f758affd640 1 -- 172.21.15.78:0/1824412472 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f75a0097300 con 0x556e3b63ad30 2024-03-16T11:08:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.696+0000 7f758affd640 1 -- 172.21.15.78:0/1824412472 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f75a0009d60 con 0x556e3b63ad30 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 10 client.25898 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556e3b380620 con 0x556e3b63ad30 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f758affd640 1 -- 172.21.15.78:0/1824412472 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75a009ac90 con 0x556e3b63ad30 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f758affd640 1 -- 172.21.15.78:0/1824412472 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f75a00d8070 con 0x556e3b63ad30 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 10 client.25898 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 10 client.25898 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 20 client.25898 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6508c0 0x556e3b652ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 4 client.25898 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f75a400fe70 con 0x556e3b6508c0 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.697+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6508c0 0x556e3b652ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.698+0000 7f758affd640 10 client.25898 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.698+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6508c0 0x556e3b652ce0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3958916d6605d046 server_cookie=11a10a7280e880e in_seq=0 out_seq=0 2024-03-16T11:08:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f758affd640 1 -- 172.21.15.78:0/1824412472 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f75a400fe70 con 0x556e3b6508c0 2024-03-16T11:08:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f758affd640 10 client.25898 handle_command_reply: tid=0 2024-03-16T11:08:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 client.25898 shutdown 2024-03-16T11:08:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 msgr2=0x556e3b644b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b644b30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f75a00970f0 tx=0x7f75a0096ff0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 shutdown_connections 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6508c0 0x556e3b652ce0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b64bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b644b30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1824412472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 0x556e3b6445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 >> 172.21.15.78:0/1824412472 conn(0x556e3b6358b0 msgr2=0x556e3b634bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f758affd640 0 client.25898 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.699+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 shutdown_connections 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.700+0000 7f75b4db6740 1 -- 172.21.15.78:0/1824412472 wait complete. 2024-03-16T11:08:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.700+0000 7f75b4db6740 20 client.25898 trim_cache size 0 max 16384 2024-03-16T11:08:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.723+0000 7f75b4db6740 1 Processor -- start 2024-03-16T11:08:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.723+0000 7f75b4db6740 1 -- start start 2024-03-16T11:08:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.723+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 0x556e3b639220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.723+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b64bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.723+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b6575e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75b4db6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1a9350 con 0x556e3b63fea0 2024-03-16T11:08:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b64bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b64bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36852/0 (socket says 172.21.15.78:36852) 2024-03-16T11:08:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 -- 172.21.15.78:0/3029108359 learned_addr learned my addr 172.21.15.78:0/3029108359 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75b4db6740 1 -- 172.21.15.78:0/3029108359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1a6c20 con 0x556e3b63ad30 2024-03-16T11:08:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75b4db6740 1 -- 172.21.15.78:0/3029108359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1a4a50 con 0x556e3b63a340 2024-03-16T11:08:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75abfff640 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b6575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 -- 172.21.15.78:0/3029108359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 msgr2=0x556e3b639220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T11:08:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 0x556e3b639220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 -- 172.21.15.78:0/3029108359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 msgr2=0x556e3b6575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b6575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 -- 172.21.15.78:0/3029108359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e3b64ee20 con 0x556e3b63ad30 2024-03-16T11:08:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.724+0000 7f75aaffd640 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b64bb30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f75a009a990 tx=0x7f75a00991a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c63ace316497792 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.725+0000 7f75917fa640 1 -- 172.21.15.78:0/3029108359 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0020040 con 0x556e3b63ad30 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.725+0000 7f75917fa640 1 -- 172.21.15.78:0/3029108359 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75a009ac90 con 0x556e3b63ad30 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.725+0000 7f75917fa640 1 -- 172.21.15.78:0/3029108359 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0097300 con 0x556e3b63ad30 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 -- 172.21.15.78:0/3029108359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 msgr2=0x556e3b64bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b64bb30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f75a009a990 tx=0x7f75a00991a0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 -- 172.21.15.78:0/3029108359 shutdown_connections 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63fea0 0x556e3b6575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b64bb30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 --2- 172.21.15.78:0/3029108359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63a340 0x556e3b639220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 -- 172.21.15.78:0/3029108359 >> 172.21.15.78:0/3029108359 conn(0x556e3b6358b0 msgr2=0x556e3b638b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 -- 172.21.15.78:0/3029108359 shutdown_connections 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.726+0000 7f75b4db6740 1 -- 172.21.15.78:0/3029108359 wait complete. 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 -- start start 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 Processor -- start 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63a340 0x556e3b649900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b649e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b64ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1a50e0 con 0x556e3b63a340 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1a6550 con 0x556e3b63ad30 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1a57b0 con 0x556e3b63fea0 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75b4db6740 10 client.0 _ll_register_callbacks cb 0x7f75a00057d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b649e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75abfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b64ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75ab7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63a340 0x556e3b649900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75abfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b64ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50718/0 (socket says 172.21.15.78:50718) 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.727+0000 7f75abfff640 1 -- 172.21.15.78:0/1691783304 learned_addr learned my addr 172.21.15.78:0/1691783304 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f75abfff640 1 -- 172.21.15.78:0/1691783304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 msgr2=0x556e3b649e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f75abfff640 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b649e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f75abfff640 1 -- 172.21.15.78:0/1691783304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63a340 msgr2=0x556e3b649900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f75abfff640 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63a340 0x556e3b649900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f75abfff640 1 -- 172.21.15.78:0/1691783304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e3b64b440 con 0x556e3b63fea0 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63a340 0x556e3b649900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f75abfff640 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b64ae50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75a401c040 tx=0x7f75a4004c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30aa6956c52aad5e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f758affd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.728+0000 7f758affd640 1 -- 172.21.15.78:0/1691783304 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4028020 con 0x556e3b63fea0 2024-03-16T11:08:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.729+0000 7f758affd640 1 -- 172.21.15.78:0/1691783304 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f75a400f9c0 con 0x556e3b63fea0 2024-03-16T11:08:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.729+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a009ac90 con 0x556e3b63fea0 2024-03-16T11:08:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.729+0000 7f758affd640 1 -- 172.21.15.78:0/1691783304 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f75a401e040 con 0x556e3b63fea0 2024-03-16T11:08:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.729+0000 7f75b4db6740 10 client.25882 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.729+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556e3b6b7320 con 0x556e3b63fea0 2024-03-16T11:08:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.729+0000 7f758affd640 1 -- 172.21.15.78:0/1691783304 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75a009ac90 con 0x556e3b63fea0 2024-03-16T11:08:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f758affd640 1 -- 172.21.15.78:0/1691783304 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f75a4005040 con 0x556e3b63fea0 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f75b4db6740 10 client.25882 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f75b4db6740 10 client.25882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f75b4db6740 20 client.25882 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6b7a80 0x556e3b6b9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f75b4db6740 4 client.25882 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e3b5acf80 con 0x556e3b6b7a80 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.731+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6b7a80 0x556e3b6b9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.732+0000 7f75ab7fe640 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6b7a80 0x556e3b6b9e70 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fcd4ad63ea4ce5f server_cookie=6ada49369691c31a in_seq=0 out_seq=0 2024-03-16T11:08:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.732+0000 7f758affd640 10 client.25882 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.746+0000 7f758affd640 1 -- 172.21.15.78:0/1691783304 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e3b5acf80 con 0x556e3b6b7a80 2024-03-16T11:08:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.746+0000 7f758affd640 10 client.25882 handle_command_reply: tid=0 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 client.25882 shutdown 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 msgr2=0x556e3b64ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b64ae50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75a401c040 tx=0x7f75a4004c30 comp rx=0 tx=0).stop 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 shutdown_connections 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x556e3b6b7a80 0x556e3b6b9e70 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e3b63fea0 0x556e3b64ae50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556e3b63ad30 0x556e3b649e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 --2- 172.21.15.78:0/1691783304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556e3b63a340 0x556e3b649900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f758affd640 0 client.25882 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 >> 172.21.15.78:0/1691783304 conn(0x556e3b6358b0 msgr2=0x556e3b6393a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:08:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:08:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:08:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:08:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 shutdown_connections 2024-03-16T11:08:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 1 -- 172.21.15.78:0/1691783304 wait complete. 2024-03-16T11:08:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.747+0000 7f75b4db6740 20 client.25882 trim_cache size 0 max 16384 2024-03-16T11:08:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75940a2930 msgr2=0x7f75940a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75940a2930 0x7f75940a4df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f759c005ab0 tx=0x7f759c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75ac1018f0 msgr2=0x7f75ac195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75ac1018f0 0x7f75ac195cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f75a000d6d0 tx=0x7f75a000db90 comp rx=0 tx=0).stop 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 shutdown_connections 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac1022b0 0x7f75ac19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75ac1018f0 0x7f75ac195cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f75940a2930 0x7f75940a4df0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 --2- 172.21.15.78:0/3061677344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac100700 0x7f75ac195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.748+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 >> 172.21.15.78:0/3061677344 conn(0x7f75ac0fbc70 msgr2=0x7f75ac104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.749+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 shutdown_connections 2024-03-16T11:08:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:11.749+0000 7f75b21d3640 1 -- 172.21.15.78:0/3061677344 wait complete. 2024-03-16T11:08:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:13 smithi098 ceph-mon[35848]: pgmap v415: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.41k op/s 2024-03-16T11:08:13.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:13 smithi153 ceph-mon[35521]: pgmap v415: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.41k op/s 2024-03-16T11:08:13.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:13 smithi078 ceph-mon[32235]: pgmap v415: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.41k op/s 2024-03-16T11:08:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:15 smithi098 ceph-mon[35848]: pgmap v416: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 19 MiB/s wr, 4.21k op/s 2024-03-16T11:08:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:15 smithi098 ceph-mon[35848]: scrub summary: idle 2024-03-16T11:08:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:15 smithi153 ceph-mon[35521]: pgmap v416: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 19 MiB/s wr, 4.21k op/s 2024-03-16T11:08:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:15 smithi153 ceph-mon[35521]: scrub summary: idle 2024-03-16T11:08:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:15 smithi078 ceph-mon[32235]: pgmap v416: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 19 MiB/s wr, 4.21k op/s 2024-03-16T11:08:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:15 smithi078 ceph-mon[32235]: scrub summary: idle 2024-03-16T11:08:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:17 smithi098 ceph-mon[35848]: pgmap v417: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.44k op/s 2024-03-16T11:08:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:17 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:08:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:17 smithi153 ceph-mon[35521]: pgmap v417: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.44k op/s 2024-03-16T11:08:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:17 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:08:17.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:17 smithi078 ceph-mon[32235]: pgmap v417: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.44k op/s 2024-03-16T11:08:17.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:17 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:08:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: pgmap v418: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 18 MiB/s wr, 4.09k op/s 2024-03-16T11:08:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:08:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:08:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:18 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: pgmap v418: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 18 MiB/s wr, 4.09k op/s 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:08:18.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:18 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:18.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: pgmap v418: 97 pgs: 97 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 18 MiB/s wr, 4.09k op/s 2024-03-16T11:08:18.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:08:18.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:08:18.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:18 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:08:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:21 smithi098 ceph-mon[35848]: pgmap v419: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:21 smithi153 ceph-mon[35521]: pgmap v419: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:21.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:21 smithi078 ceph-mon[32235]: pgmap v419: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:23 smithi098 ceph-mon[35848]: pgmap v420: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.34k op/s 2024-03-16T11:08:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:23 smithi153 ceph-mon[35521]: pgmap v420: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.34k op/s 2024-03-16T11:08:23.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:23 smithi078 ceph-mon[32235]: pgmap v420: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 MiB/s wr, 4.34k op/s 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 -- 172.21.15.78:0/2467648945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201022b0 msgr2=0x7ff820108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 --2- 172.21.15.78:0/2467648945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201022b0 0x7ff820108b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff818009ca0 tx=0x7ff81802c100 comp rx=0 tx=0).stop 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 -- 172.21.15.78:0/2467648945 shutdown_connections 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 --2- 172.21.15.78:0/2467648945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201022b0 0x7ff820108b40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 --2- 172.21.15.78:0/2467648945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8201018f0 0x7ff820101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 --2- 172.21.15.78:0/2467648945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 0x7ff820100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 -- 172.21.15.78:0/2467648945 >> 172.21.15.78:0/2467648945 conn(0x7ff8200fbc70 msgr2=0x7ff8200fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 -- 172.21.15.78:0/2467648945 shutdown_connections 2024-03-16T11:08:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.883+0000 7ff827826640 1 -- 172.21.15.78:0/2467648945 wait complete. 2024-03-16T11:08:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.884+0000 7ff827826640 1 Processor -- start 2024-03-16T11:08:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.884+0000 7ff827826640 1 -- start start 2024-03-16T11:08:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.884+0000 7ff827826640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 0x7ff82019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.884+0000 7ff827826640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201018f0 0x7ff82019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.884+0000 7ff827826640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8201022b0 0x7ff8201a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff827826640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff82010b0f0 con 0x7ff820100700 2024-03-16T11:08:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff827826640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff82010af70 con 0x7ff8201022b0 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff827826640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff82010b270 con 0x7ff8201018f0 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 0x7ff82019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff825dc3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8201022b0 0x7ff8201a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 0x7ff82019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43902/0 (socket says 172.21.15.78:43902) 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 -- 172.21.15.78:0/1754234297 learned_addr learned my addr 172.21.15.78:0/1754234297 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff824dc1640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201018f0 0x7ff82019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 -- 172.21.15.78:0/1754234297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201018f0 msgr2=0x7ff82019e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201018f0 0x7ff82019e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 -- 172.21.15.78:0/1754234297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8201022b0 msgr2=0x7ff8201a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8201022b0 0x7ff8201a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 -- 172.21.15.78:0/1754234297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8201a5b50 con 0x7ff820100700 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff824dc1640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201018f0 0x7ff82019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.885+0000 7ff8255c2640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 0x7ff82019df30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00e9c0 tx=0x7ff81c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.886+0000 7ff8167fc640 1 -- 172.21.15.78:0/1754234297 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c00cd80 con 0x7ff820100700 2024-03-16T11:08:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.886+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8201a5e40 con 0x7ff820100700 2024-03-16T11:08:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.886+0000 7ff8167fc640 1 -- 172.21.15.78:0/1754234297 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff81c0035f0 con 0x7ff820100700 2024-03-16T11:08:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.886+0000 7ff8167fc640 1 -- 172.21.15.78:0/1754234297 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c010690 con 0x7ff820100700 2024-03-16T11:08:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.886+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8201a6300 con 0x7ff820100700 2024-03-16T11:08:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.889+0000 7ff8167fc640 1 -- 172.21.15.78:0/1754234297 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ff81c0037a0 con 0x7ff820100700 2024-03-16T11:08:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.889+0000 7ff82a40b740 1 Processor -- start 2024-03-16T11:08:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff82a40b740 1 -- start start 2024-03-16T11:08:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1842760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd1847830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd184a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff82a40b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55fcd17a7a40 con 0x55fcd1847ea0 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff82a40b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fcd1750920 con 0x55fcd1842d30 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff82a40b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcd17a3ae0 con 0x55fcd1842340 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff825dc3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1842760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff825dc3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1842760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:55190/0 (socket says 172.21.15.78:55190) 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff825dc3640 1 -- 172.21.15.78:0/3633402044 learned_addr learned my addr 172.21.15.78:0/3633402044 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff824dc1640 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd184a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff8255c2640 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd1847830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff825dc3640 1 -- 172.21.15.78:0/3633402044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 msgr2=0x55fcd1847830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff825dc3640 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd1847830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.890+0000 7ff825dc3640 1 -- 172.21.15.78:0/3633402044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 msgr2=0x55fcd184a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.891+0000 7ff825dc3640 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd184a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.891+0000 7ff825dc3640 1 -- 172.21.15.78:0/3633402044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd1768460 con 0x55fcd1842340 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.891+0000 7ff825dc3640 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1842760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff81802c710 tx=0x7ff818009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aa4e2a633a62b9e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.891+0000 7ff8167fc640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff80c0a29a0 0x7ff80c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.891+0000 7ff8255c2640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff80c0a29a0 0x7ff80c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.891+0000 7ff80b7fe640 1 -- 172.21.15.78:0/3633402044 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818043070 con 0x55fcd1842340 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.892+0000 7ff80b7fe640 1 -- 172.21.15.78:0/3633402044 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff818003b70 con 0x55fcd1842340 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.892+0000 7ff80b7fe640 1 -- 172.21.15.78:0/3633402044 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818039040 con 0x55fcd1842340 2024-03-16T11:08:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.892+0000 7ff8167fc640 1 -- 172.21.15.78:0/1754234297 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff81c0cee60 con 0x7ff820100700 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff82a40b740 1 -- 172.21.15.78:0/3633402044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 msgr2=0x55fcd1842760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1842760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff81802c710 tx=0x7ff818009800 comp rx=0 tx=0).stop 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff8255c2640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff80c0a29a0 0x7ff80c0a4e60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff81c097c20 tx=0x7ff81c0d2820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff82a40b740 1 -- 172.21.15.78:0/3633402044 shutdown_connections 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd184a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd1847830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3633402044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1842760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.893+0000 7ff82a40b740 1 -- 172.21.15.78:0/3633402044 >> 172.21.15.78:0/3633402044 conn(0x55fcd183d8b0 msgr2=0x55fcd183dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:24.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.894+0000 7ff82a40b740 1 -- 172.21.15.78:0/3633402044 shutdown_connections 2024-03-16T11:08:24.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.894+0000 7ff82a40b740 1 -- 172.21.15.78:0/3633402044 wait complete. 2024-03-16T11:08:24.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.895+0000 7ff82a40b740 1 -- start start 2024-03-16T11:08:24.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.895+0000 7ff82a40b740 1 Processor -- start 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd185d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd185dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd1864c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff825dc3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd185d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff825dc3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd185d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:55200/0 (socket says 172.21.15.78:55200) 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff825dc3640 1 -- 172.21.15.78:0/116575738 learned_addr learned my addr 172.21.15.78:0/116575738 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55fcd185be00 con 0x55fcd1847ea0 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fcd1763d70 con 0x55fcd1842d30 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcd185bf80 con 0x55fcd1842340 2024-03-16T11:08:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.896+0000 7ff824dc1640 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd185dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff82a40b740 10 client.0 _ll_register_callbacks cb 0x55fcd14bba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff825dc3640 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd1864c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff825dc3640 1 -- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 msgr2=0x55fcd185dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff825dc3640 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd185dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff825dc3640 1 -- 172.21.15.78:0/116575738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 msgr2=0x55fcd1864c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff825dc3640 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd1864c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff825dc3640 1 -- 172.21.15.78:0/116575738 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd14bc210 con 0x55fcd1842340 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff818003b70 con 0x55fcd1842340 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff825dc3640 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd185d680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff818032520 tx=0x7ff81802c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f2158366fc9c19d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.897+0000 7ff808ff9640 10 client.25894 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.898+0000 7ff808ff9640 1 -- 172.21.15.78:0/116575738 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818059050 con 0x55fcd1842340 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.898+0000 7ff808ff9640 1 -- 172.21.15.78:0/116575738 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff818003d20 con 0x55fcd1842340 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.898+0000 7ff808ff9640 1 -- 172.21.15.78:0/116575738 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff818039040 con 0x55fcd1842340 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.898+0000 7ff82a40b740 10 client.25894 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.898+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55fcd1588620 con 0x55fcd1842340 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.898+0000 7ff808ff9640 1 -- 172.21.15.78:0/116575738 <== mon.1 v2:172.21.15.153:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff81805c060 con 0x55fcd1842340 2024-03-16T11:08:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.899+0000 7ff82a40b740 10 client.25894 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.899+0000 7ff82a40b740 10 client.25894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.899+0000 7ff82a40b740 20 client.25894 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.899+0000 7ff82a40b740 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd186b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.899+0000 7ff82a40b740 4 client.25894 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.899+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcd17b4f80 con 0x55fcd1868f70 2024-03-16T11:08:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.900+0000 7ff824dc1640 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd186b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.900+0000 7ff824dc1640 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd186b360 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6b2606f96ecbdea server_cookie=84b668973441d452 in_seq=0 out_seq=0 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.900+0000 7ff808ff9640 10 client.25894 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff808ff9640 1 -- 172.21.15.78:0/116575738 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcd17b4f80 con 0x55fcd1868f70 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff808ff9640 10 client.25894 handle_command_reply: tid=0 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff82a40b740 1 client.25894 shutdown 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 msgr2=0x55fcd185d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff82a40b740 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd185d680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff818032520 tx=0x7ff81802c100 comp rx=0 tx=0).stop 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 shutdown_connections 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff82a40b740 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd186b360 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.901+0000 7ff82a40b740 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1847ea0 0x55fcd1864c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.902+0000 7ff808ff9640 0 client.25894 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.902+0000 7ff82a40b740 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1842d30 0x55fcd185dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.902+0000 7ff82a40b740 1 --2- 172.21.15.78:0/116575738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd185d680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.902+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 >> 172.21.15.78:0/116575738 conn(0x55fcd183d8b0 msgr2=0x55fcd183cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.902+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 shutdown_connections 2024-03-16T11:08:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.902+0000 7ff82a40b740 1 -- 172.21.15.78:0/116575738 wait complete. 2024-03-16T11:08:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.902+0000 7ff82a40b740 20 client.25894 trim_cache size 0 max 16384 2024-03-16T11:08:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.928+0000 7ff82a40b740 1 Processor -- start 2024-03-16T11:08:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff82a40b740 1 -- start start 2024-03-16T11:08:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1870a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842d30 0x55fcd1864b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd1840b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff82a40b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff8201a9300 con 0x55fcd1842d30 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff82a40b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8201a6bd0 con 0x55fcd1847ea0 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff82a40b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff82019c0f0 con 0x55fcd1842340 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff824dc1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842d30 0x55fcd1864b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff824dc1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842d30 0x55fcd1864b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43932/0 (socket says 172.21.15.78:43932) 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff824dc1640 1 -- 172.21.15.78:0/1155307228 learned_addr learned my addr 172.21.15.78:0/1155307228 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff825dc3640 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1870a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.929+0000 7ff825dc3640 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd1840b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff824dc1640 1 -- 172.21.15.78:0/1155307228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 msgr2=0x55fcd1870a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff824dc1640 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1870a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff824dc1640 1 -- 172.21.15.78:0/1155307228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 msgr2=0x55fcd1840b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff824dc1640 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd1840b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff824dc1640 1 -- 172.21.15.78:0/1155307228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd183e620 con 0x55fcd1842d30 2024-03-16T11:08:24.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff824dc1640 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842d30 0x55fcd1864b90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55fcd183e820 tx=0x7ff810005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b2c9b29ae5ff7f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff80affd640 1 -- 172.21.15.78:0/1155307228 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100212e0 con 0x55fcd1842d30 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff80affd640 1 -- 172.21.15.78:0/1155307228 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff810025070 con 0x55fcd1842d30 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.930+0000 7ff80affd640 1 -- 172.21.15.78:0/1155307228 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81001fe20 con 0x55fcd1842d30 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.931+0000 7ff82a40b740 1 -- 172.21.15.78:0/1155307228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842d30 msgr2=0x55fcd1864b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.931+0000 7ff82a40b740 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842d30 0x55fcd1864b90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55fcd183e820 tx=0x7ff810005f90 comp rx=0 tx=0).stop 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.931+0000 7ff82a40b740 1 -- 172.21.15.78:0/1155307228 shutdown_connections 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.931+0000 7ff82a40b740 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd1840b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.931+0000 7ff82a40b740 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842d30 0x55fcd1864b90 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.931+0000 7ff82a40b740 1 --2- 172.21.15.78:0/1155307228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842340 0x55fcd1870a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.931+0000 7ff82a40b740 1 -- 172.21.15.78:0/1155307228 >> 172.21.15.78:0/1155307228 conn(0x55fcd183d8b0 msgr2=0x55fcd1838a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.932+0000 7ff82a40b740 1 -- 172.21.15.78:0/1155307228 shutdown_connections 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.932+0000 7ff82a40b740 1 -- 172.21.15.78:0/1155307228 wait complete. 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.932+0000 7ff82a40b740 1 -- start start 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.932+0000 7ff82a40b740 1 Processor -- start 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842340 0x55fcd18623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842d30 0x55fcd18628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff82a40b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd18637a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff82a40b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff82019c710 con 0x55fcd1842340 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff82a40b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8201a65c0 con 0x55fcd1847ea0 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff82a40b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff82019cde0 con 0x55fcd1842d30 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff82a40b740 10 client.0 _ll_register_callbacks cb 0x7ff8100095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842340 0x55fcd18623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842340 0x55fcd18623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43936/0 (socket says 172.21.15.78:43936) 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff824dc1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842d30 0x55fcd18628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 -- 172.21.15.78:0/3058707196 learned_addr learned my addr 172.21.15.78:0/3058707196 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd18637a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 -- 172.21.15.78:0/3058707196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842d30 msgr2=0x55fcd18628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842d30 0x55fcd18628e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 -- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 msgr2=0x55fcd18637a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd18637a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.933+0000 7ff825dc3640 1 -- 172.21.15.78:0/3058707196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd185bd40 con 0x55fcd1842340 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff825dc3640 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842340 0x55fcd18623a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff818005e30 tx=0x7ff81804a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2f6ddc88efa397 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff808ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff808ff9640 1 -- 172.21.15.78:0/3058707196 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818059070 con 0x55fcd1842340 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff808ff9640 1 -- 172.21.15.78:0/3058707196 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff818032d00 con 0x55fcd1842340 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff810025070 con 0x55fcd1842340 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff808ff9640 1 -- 172.21.15.78:0/3058707196 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff818062040 con 0x55fcd1842340 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff82a40b740 10 client.16092 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55fcd18cf050 con 0x55fcd1842340 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.934+0000 7ff808ff9640 1 -- 172.21.15.78:0/3058707196 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff818003b70 con 0x55fcd1842340 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff808ff9640 1 -- 172.21.15.78:0/3058707196 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff818052070 con 0x55fcd1842340 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff82a40b740 10 client.16092 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff82a40b740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff82a40b740 20 client.16092 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd18d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff82a40b740 4 client.16092 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff810025070 con 0x55fcd1868f70 2024-03-16T11:08:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.935+0000 7ff824dc1640 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd18d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:24.939 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.936+0000 7ff824dc1640 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd18d1860 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d8fc6281df6612f server_cookie=e79f37636d61acfc in_seq=0 out_seq=0 2024-03-16T11:08:24.939 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.936+0000 7ff808ff9640 10 client.16092 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff808ff9640 1 -- 172.21.15.78:0/3058707196 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff810025070 con 0x55fcd1868f70 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff808ff9640 10 client.16092 handle_command_reply: tid=0 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 client.16092 shutdown 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842340 msgr2=0x55fcd18623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842340 0x55fcd18623a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff818005e30 tx=0x7ff81804a040 comp rx=0 tx=0).stop 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 shutdown_connections 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x55fcd1868f70 0x55fcd18d1860 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fcd1847ea0 0x55fcd18637a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff808ff9640 0 client.16092 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcd1842d30 0x55fcd18628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 --2- 172.21.15.78:0/3058707196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fcd1842340 0x55fcd18623a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 >> 172.21.15.78:0/3058707196 conn(0x55fcd183d8b0 msgr2=0x55fcd1840b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 shutdown_connections 2024-03-16T11:08:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 1 -- 172.21.15.78:0/3058707196 wait complete. 2024-03-16T11:08:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.949+0000 7ff82a40b740 20 client.16092 trim_cache size 0 max 16384 2024-03-16T11:08:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-03-16T11:08:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-03-16T11:08:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-03-16T11:08:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-03-16T11:08:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff80c0a29a0 msgr2=0x7ff80c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff80c0a29a0 0x7ff80c0a4e60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff81c097c20 tx=0x7ff81c0d2820 comp rx=0 tx=0).stop 2024-03-16T11:08:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 msgr2=0x7ff82019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 0x7ff82019df30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00e9c0 tx=0x7ff81c00ee80 comp rx=0 tx=0).stop 2024-03-16T11:08:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 shutdown_connections 2024-03-16T11:08:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8201022b0 0x7ff8201a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff80c0a29a0 0x7ff80c0a4e60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8201018f0 0x7ff82019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 --2- 172.21.15.78:0/1754234297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff820100700 0x7ff82019df30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.950+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 >> 172.21.15.78:0/1754234297 conn(0x7ff8200fbc70 msgr2=0x7ff8200fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.952+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 shutdown_connections 2024-03-16T11:08:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-03-16T11:08:24.952+0000 7ff827826640 1 -- 172.21.15.78:0/1754234297 wait complete. 2024-03-16T11:08:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:25 smithi098 ceph-mon[35848]: pgmap v421: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 18 MiB/s wr, 4.16k op/s 2024-03-16T11:08:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:25 smithi153 ceph-mon[35521]: pgmap v421: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 18 MiB/s wr, 4.16k op/s 2024-03-16T11:08:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:25 smithi078 ceph-mon[32235]: pgmap v421: 97 pgs: 97 active+clean; 258 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 18 MiB/s wr, 4.16k op/s 2024-03-16T11:08:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:27 smithi098 ceph-mon[35848]: pgmap v422: 97 pgs: 97 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 4.41k op/s 2024-03-16T11:08:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:27 smithi153 ceph-mon[35521]: pgmap v422: 97 pgs: 97 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 4.41k op/s 2024-03-16T11:08:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:27 smithi078 ceph-mon[32235]: pgmap v422: 97 pgs: 97 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 4.41k op/s 2024-03-16T11:08:29.432 INFO:tasks.workunit.client.0.smithi078.stdout:setting up random write file 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:done setting up random write file 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:starting fsync run 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0045s fsync time: 0.0060s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0024s fsync time: 0.0381s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0022s fsync time: 0.0553s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0021s fsync time: 0.0404s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0027s fsync time: 0.0345s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0029s fsync time: 0.0446s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0036s fsync time: 0.0382s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0045s fsync time: 0.0458s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0049s fsync time: 0.0722s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0038s fsync time: 0.0479s 2024-03-16T11:08:29.433 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0022s fsync time: 0.0279s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0392s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0025s fsync time: 0.0568s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0027s fsync time: 0.0328s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0050s fsync time: 0.0374s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0025s fsync time: 0.0538s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0026s fsync time: 0.0341s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0398s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0024s fsync time: 0.0502s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0021s fsync time: 0.0234s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0021s fsync time: 0.0388s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0025s fsync time: 0.0653s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0032s fsync time: 0.0256s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0025s fsync time: 0.0587s 2024-03-16T11:08:29.434 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0028s fsync time: 0.0230s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0026s fsync time: 0.0278s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0022s fsync time: 0.0703s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0026s fsync time: 0.0295s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0024s fsync time: 0.0295s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0025s fsync time: 0.0605s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0030s fsync time: 0.0491s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0425s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0030s fsync time: 0.0641s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0021s fsync time: 0.0358s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0024s fsync time: 0.0645s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0027s fsync time: 0.0531s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0401s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0035s fsync time: 0.0366s 2024-03-16T11:08:29.435 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0075s fsync time: 0.0383s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0036s fsync time: 0.0562s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0032s fsync time: 0.0333s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0044s fsync time: 0.0271s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0028s fsync time: 0.0326s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0039s fsync time: 0.0589s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0335s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0473s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0029s fsync time: 0.0246s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0127s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0027s fsync time: 0.0423s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0024s fsync time: 0.0313s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0022s fsync time: 0.0462s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0371s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0017s fsync time: 0.0260s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0030s fsync time: 0.0271s 2024-03-16T11:08:29.436 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0023s fsync time: 0.0295s 2024-03-16T11:08:29.437 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0017s fsync time: 0.0476s 2024-03-16T11:08:29.437 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0027s fsync time: 0.0324s 2024-03-16T11:08:29.437 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0030s fsync time: 0.0429s 2024-03-16T11:08:29.437 INFO:tasks.workunit.client.0.smithi078.stdout:write time: 0.0027s fsync time: 0.0322s 2024-03-16T11:08:29.437 INFO:tasks.workunit.client.0.smithi078.stdout:run done 59 fsyncs total, killing random writer 2024-03-16T11:08:29.439 INFO:tasks.workunit.client.0.smithi078.stdout:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-16T11:08:29.440 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-16T11:08:29.440 INFO:tasks.workunit.client.0.smithi078.stderr:+ whereis lsof 2024-03-16T11:08:29.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:29 smithi098 ceph-mon[35848]: pgmap v423: 97 pgs: 97 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 18 MiB/s wr, 4.09k op/s 2024-03-16T11:08:29.470 INFO:tasks.workunit.client.0.smithi078.stdout:lsof: /usr/bin/lsof /usr/share/man/man1/lsof.1.gz 2024-03-16T11:08:29.470 INFO:tasks.workunit.client.0.smithi078.stderr:+ lsof 2024-03-16T11:08:29.492 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/423aaf3f97e17b1d0cc1a5e5e051e9e2ba53f4272f504e98c8d7863e8506e480/merged 2024-03-16T11:08:29.492 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.492 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/fb1e8e3fd00a93af367d948525932bcbcc81edf80f277f86811cc963f874353c/merged 2024-03-16T11:08:29.492 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.492 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/26cc722b26f84079a7098fe9997b196f203186181bfb5d054409701fbe904409/merged 2024-03-16T11:08:29.492 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ef20c091f21d7de64efe896d862b46d8cbc6d6d7a4e0bfda8f4a371dda535b4d/merged 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/b3c4b210ea5d8c49071bae38fc56fb7641cc866bbc0838b8cec727b70e68fa11/merged 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/2b1baf3368514f29dfd119893f2641f932dedc14b86c47776779f1880da608ad/merged 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/618609b1a80cc24f311fa2a9cf028a3077a47eefa0261d7a96cf20ff62e9cd49/merged 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/9027617660d0714387e5439cbc459c21940c431e0df8ee0c8cb3c45e6eb86b5e/merged 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/f08ec27e25346589231af78b449b354ab6fe60d3dcdaf87fe339af67b1e29cb9/merged 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/899e01c2739b0eb78ede970477dfe22716fbd9455d9e267e239e2312cc94de67/merged 2024-03-16T11:08:29.493 INFO:tasks.workunit.client.0.smithi078.stderr: Output information may be incomplete. 2024-03-16T11:08:29.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:29 smithi153 ceph-mon[35521]: pgmap v423: 97 pgs: 97 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 18 MiB/s wr, 4.09k op/s 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 1 root cwd unknown /proc/1/cwd (readlink: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 1 root rtd unknown /proc/1/root (readlink: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 1 root txt unknown /proc/1/exe (readlink: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 1 root NOFD /proc/1/fd (opendir: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:kthreadd 2 root cwd unknown /proc/2/cwd (readlink: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:kthreadd 2 root rtd unknown /proc/2/root (readlink: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:kthreadd 2 root txt unknown /proc/2/exe (readlink: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:kthreadd 2 root NOFD /proc/2/fd (opendir: Permission denied) 2024-03-16T11:08:29.552 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_gp 3 root cwd unknown /proc/3/cwd (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_gp 3 root rtd unknown /proc/3/root (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_gp 3 root txt unknown /proc/3/exe (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_gp 3 root NOFD /proc/3/fd (opendir: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_par_g 4 root cwd unknown /proc/4/cwd (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_par_g 4 root rtd unknown /proc/4/root (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_par_g 4 root txt unknown /proc/4/exe (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_par_g 4 root NOFD /proc/4/fd (opendir: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:slub_flus 5 root cwd unknown /proc/5/cwd (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:slub_flus 5 root rtd unknown /proc/5/root (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:slub_flus 5 root txt unknown /proc/5/exe (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:slub_flus 5 root NOFD /proc/5/fd (opendir: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:netns 6 root cwd unknown /proc/6/cwd (readlink: Permission denied) 2024-03-16T11:08:29.553 INFO:tasks.workunit.client.0.smithi078.stdout:netns 6 root rtd unknown /proc/6/root (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:netns 6 root txt unknown /proc/6/exe (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:netns 6 root NOFD /proc/6/fd (opendir: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 8 root cwd unknown /proc/8/cwd (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 8 root rtd unknown /proc/8/root (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 8 root txt unknown /proc/8/exe (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 8 root NOFD /proc/8/fd (opendir: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:mm_percpu 10 root cwd unknown /proc/10/cwd (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:mm_percpu 10 root rtd unknown /proc/10/root (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:mm_percpu 10 root txt unknown /proc/10/exe (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:mm_percpu 10 root NOFD /proc/10/fd (opendir: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 12 root cwd unknown /proc/12/cwd (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 12 root rtd unknown /proc/12/root (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 12 root txt unknown /proc/12/exe (readlink: Permission denied) 2024-03-16T11:08:29.554 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 12 root NOFD /proc/12/fd (opendir: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 13 root cwd unknown /proc/13/cwd (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 13 root rtd unknown /proc/13/root (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 13 root txt unknown /proc/13/exe (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 13 root NOFD /proc/13/fd (opendir: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 14 root cwd unknown /proc/14/cwd (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 14 root rtd unknown /proc/14/root (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 14 root txt unknown /proc/14/exe (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_tasks 14 root NOFD /proc/14/fd (opendir: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 15 root cwd unknown /proc/15/cwd (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 15 root rtd unknown /proc/15/root (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 15 root txt unknown /proc/15/exe (readlink: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 15 root NOFD /proc/15/fd (opendir: Permission denied) 2024-03-16T11:08:29.555 INFO:tasks.workunit.client.0.smithi078.stdout:pr/ttyS1 16 root cwd unknown /proc/16/cwd (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:pr/ttyS1 16 root rtd unknown /proc/16/root (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:pr/ttyS1 16 root txt unknown /proc/16/exe (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:pr/ttyS1 16 root NOFD /proc/16/fd (opendir: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_preem 17 root cwd unknown /proc/17/cwd (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_preem 17 root rtd unknown /proc/17/root (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_preem 17 root txt unknown /proc/17/exe (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:rcu_preem 17 root NOFD /proc/17/fd (opendir: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:migration 18 root cwd unknown /proc/18/cwd (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:migration 18 root rtd unknown /proc/18/root (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:migration 18 root txt unknown /proc/18/exe (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:migration 18 root NOFD /proc/18/fd (opendir: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 19 root cwd unknown /proc/19/cwd (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 19 root rtd unknown /proc/19/root (readlink: Permission denied) 2024-03-16T11:08:29.556 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 19 root txt unknown /proc/19/exe (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 19 root NOFD /proc/19/fd (opendir: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/0 21 root cwd unknown /proc/21/cwd (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/0 21 root rtd unknown /proc/21/root (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/0 21 root txt unknown /proc/21/exe (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/0 21 root NOFD /proc/21/fd (opendir: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/1 22 root cwd unknown /proc/22/cwd (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/1 22 root rtd unknown /proc/22/root (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/1 22 root txt unknown /proc/22/exe (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/1 22 root NOFD /proc/22/fd (opendir: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 23 root cwd unknown /proc/23/cwd (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 23 root rtd unknown /proc/23/root (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 23 root txt unknown /proc/23/exe (readlink: Permission denied) 2024-03-16T11:08:29.557 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 23 root NOFD /proc/23/fd (opendir: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:migration 24 root cwd unknown /proc/24/cwd (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:migration 24 root rtd unknown /proc/24/root (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:migration 24 root txt unknown /proc/24/exe (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:migration 24 root NOFD /proc/24/fd (opendir: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 25 root cwd unknown /proc/25/cwd (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 25 root rtd unknown /proc/25/root (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 25 root txt unknown /proc/25/exe (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 25 root NOFD /proc/25/fd (opendir: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 27 root cwd unknown /proc/27/cwd (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 27 root rtd unknown /proc/27/root (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 27 root txt unknown /proc/27/exe (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 27 root NOFD /proc/27/fd (opendir: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/2 28 root cwd unknown /proc/28/cwd (readlink: Permission denied) 2024-03-16T11:08:29.558 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/2 28 root rtd unknown /proc/28/root (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/2 28 root txt unknown /proc/28/exe (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/2 28 root NOFD /proc/28/fd (opendir: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 29 root cwd unknown /proc/29/cwd (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 29 root rtd unknown /proc/29/root (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 29 root txt unknown /proc/29/exe (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 29 root NOFD /proc/29/fd (opendir: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:migration 30 root cwd unknown /proc/30/cwd (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:migration 30 root rtd unknown /proc/30/root (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:migration 30 root txt unknown /proc/30/exe (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:migration 30 root NOFD /proc/30/fd (opendir: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 31 root cwd unknown /proc/31/cwd (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 31 root rtd unknown /proc/31/root (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 31 root txt unknown /proc/31/exe (readlink: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 31 root NOFD /proc/31/fd (opendir: Permission denied) 2024-03-16T11:08:29.559 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32 root cwd unknown /proc/32/cwd (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32 root rtd unknown /proc/32/root (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32 root txt unknown /proc/32/exe (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32 root NOFD /proc/32/fd (opendir: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 33 root cwd unknown /proc/33/cwd (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 33 root rtd unknown /proc/33/root (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 33 root txt unknown /proc/33/exe (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 33 root NOFD /proc/33/fd (opendir: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/3 34 root cwd unknown /proc/34/cwd (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/3 34 root rtd unknown /proc/34/root (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/3 34 root txt unknown /proc/34/exe (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/3 34 root NOFD /proc/34/fd (opendir: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 35 root cwd unknown /proc/35/cwd (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 35 root rtd unknown /proc/35/root (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 35 root txt unknown /proc/35/exe (readlink: Permission denied) 2024-03-16T11:08:29.560 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 35 root NOFD /proc/35/fd (opendir: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:migration 36 root cwd unknown /proc/36/cwd (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:migration 36 root rtd unknown /proc/36/root (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:migration 36 root txt unknown /proc/36/exe (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:migration 36 root NOFD /proc/36/fd (opendir: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 37 root cwd unknown /proc/37/cwd (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 37 root rtd unknown /proc/37/root (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 37 root txt unknown /proc/37/exe (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 37 root NOFD /proc/37/fd (opendir: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 39 root cwd unknown /proc/39/cwd (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 39 root rtd unknown /proc/39/root (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 39 root txt unknown /proc/39/exe (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 39 root NOFD /proc/39/fd (opendir: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/4 40 root cwd unknown /proc/40/cwd (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/4 40 root rtd unknown /proc/40/root (readlink: Permission denied) 2024-03-16T11:08:29.561 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/4 40 root txt unknown /proc/40/exe (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/4 40 root NOFD /proc/40/fd (opendir: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 41 root cwd unknown /proc/41/cwd (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 41 root rtd unknown /proc/41/root (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 41 root txt unknown /proc/41/exe (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 41 root NOFD /proc/41/fd (opendir: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:migration 42 root cwd unknown /proc/42/cwd (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:migration 42 root rtd unknown /proc/42/root (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:migration 42 root txt unknown /proc/42/exe (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:migration 42 root NOFD /proc/42/fd (opendir: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 43 root cwd unknown /proc/43/cwd (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 43 root rtd unknown /proc/43/root (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 43 root txt unknown /proc/43/exe (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 43 root NOFD /proc/43/fd (opendir: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 45 root cwd unknown /proc/45/cwd (readlink: Permission denied) 2024-03-16T11:08:29.562 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 45 root rtd unknown /proc/45/root (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 45 root txt unknown /proc/45/exe (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 45 root NOFD /proc/45/fd (opendir: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/5 46 root cwd unknown /proc/46/cwd (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/5 46 root rtd unknown /proc/46/root (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/5 46 root txt unknown /proc/46/exe (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/5 46 root NOFD /proc/46/fd (opendir: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 47 root cwd unknown /proc/47/cwd (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 47 root rtd unknown /proc/47/root (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 47 root txt unknown /proc/47/exe (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 47 root NOFD /proc/47/fd (opendir: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:migration 48 root cwd unknown /proc/48/cwd (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:migration 48 root rtd unknown /proc/48/root (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:migration 48 root txt unknown /proc/48/exe (readlink: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:migration 48 root NOFD /proc/48/fd (opendir: Permission denied) 2024-03-16T11:08:29.563 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 49 root cwd unknown /proc/49/cwd (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 49 root rtd unknown /proc/49/root (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 49 root txt unknown /proc/49/exe (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 49 root NOFD /proc/49/fd (opendir: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 51 root cwd unknown /proc/51/cwd (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 51 root rtd unknown /proc/51/root (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 51 root txt unknown /proc/51/exe (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 51 root NOFD /proc/51/fd (opendir: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/6 52 root cwd unknown /proc/52/cwd (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/6 52 root rtd unknown /proc/52/root (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/6 52 root txt unknown /proc/52/exe (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/6 52 root NOFD /proc/52/fd (opendir: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 53 root cwd unknown /proc/53/cwd (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 53 root rtd unknown /proc/53/root (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 53 root txt unknown /proc/53/exe (readlink: Permission denied) 2024-03-16T11:08:29.564 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 53 root NOFD /proc/53/fd (opendir: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:migration 54 root cwd unknown /proc/54/cwd (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:migration 54 root rtd unknown /proc/54/root (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:migration 54 root txt unknown /proc/54/exe (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:migration 54 root NOFD /proc/54/fd (opendir: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 55 root cwd unknown /proc/55/cwd (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 55 root rtd unknown /proc/55/root (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 55 root txt unknown /proc/55/exe (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 55 root NOFD /proc/55/fd (opendir: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 57 root cwd unknown /proc/57/cwd (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 57 root rtd unknown /proc/57/root (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 57 root txt unknown /proc/57/exe (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 57 root NOFD /proc/57/fd (opendir: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/7 58 root cwd unknown /proc/58/cwd (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/7 58 root rtd unknown /proc/58/root (readlink: Permission denied) 2024-03-16T11:08:29.565 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/7 58 root txt unknown /proc/58/exe (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:cpuhp/7 58 root NOFD /proc/58/fd (opendir: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 59 root cwd unknown /proc/59/cwd (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 59 root rtd unknown /proc/59/root (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 59 root txt unknown /proc/59/exe (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:idle_inje 59 root NOFD /proc/59/fd (opendir: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:migration 60 root cwd unknown /proc/60/cwd (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:migration 60 root rtd unknown /proc/60/root (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:migration 60 root txt unknown /proc/60/exe (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:migration 60 root NOFD /proc/60/fd (opendir: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 61 root cwd unknown /proc/61/cwd (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 61 root rtd unknown /proc/61/root (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 61 root txt unknown /proc/61/exe (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:ksoftirqd 61 root NOFD /proc/61/fd (opendir: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 63 root cwd unknown /proc/63/cwd (readlink: Permission denied) 2024-03-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 63 root rtd unknown /proc/63/root (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 63 root txt unknown /proc/63/exe (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 63 root NOFD /proc/63/fd (opendir: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 66 root cwd unknown /proc/66/cwd (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 66 root rtd unknown /proc/66/root (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 66 root txt unknown /proc/66/exe (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 66 root NOFD /proc/66/fd (opendir: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 67 root cwd unknown /proc/67/cwd (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 67 root rtd unknown /proc/67/root (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 67 root txt unknown /proc/67/exe (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 67 root NOFD /proc/67/fd (opendir: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 70 root cwd unknown /proc/70/cwd (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 70 root rtd unknown /proc/70/root (readlink: Permission denied) 2024-03-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 70 root txt unknown /proc/70/exe (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 70 root NOFD /proc/70/fd (opendir: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kdevtmpfs 71 root cwd unknown /proc/71/cwd (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kdevtmpfs 71 root rtd unknown /proc/71/root (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kdevtmpfs 71 root txt unknown /proc/71/exe (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kdevtmpfs 71 root NOFD /proc/71/fd (opendir: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:inet_frag 72 root cwd unknown /proc/72/cwd (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:inet_frag 72 root rtd unknown /proc/72/root (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:inet_frag 72 root txt unknown /proc/72/exe (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:inet_frag 72 root NOFD /proc/72/fd (opendir: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kauditd 73 root cwd unknown /proc/73/cwd (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kauditd 73 root rtd unknown /proc/73/root (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kauditd 73 root txt unknown /proc/73/exe (readlink: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:kauditd 73 root NOFD /proc/73/fd (opendir: Permission denied) 2024-03-16T11:08:29.568 INFO:tasks.workunit.client.0.smithi078.stdout:khungtask 75 root cwd unknown /proc/75/cwd (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:khungtask 75 root rtd unknown /proc/75/root (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:khungtask 75 root txt unknown /proc/75/exe (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:khungtask 75 root NOFD /proc/75/fd (opendir: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:oom_reape 76 root cwd unknown /proc/76/cwd (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:oom_reape 76 root rtd unknown /proc/76/root (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:oom_reape 76 root txt unknown /proc/76/exe (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:oom_reape 76 root NOFD /proc/76/fd (opendir: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:writeback 77 root cwd unknown /proc/77/cwd (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:writeback 77 root rtd unknown /proc/77/root (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:writeback 77 root txt unknown /proc/77/exe (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:writeback 77 root NOFD /proc/77/fd (opendir: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:kcompactd 78 root cwd unknown /proc/78/cwd (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:kcompactd 78 root rtd unknown /proc/78/root (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:kcompactd 78 root txt unknown /proc/78/exe (readlink: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:kcompactd 78 root NOFD /proc/78/fd (opendir: Permission denied) 2024-03-16T11:08:29.569 INFO:tasks.workunit.client.0.smithi078.stdout:ksmd 79 root cwd unknown /proc/79/cwd (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:ksmd 79 root rtd unknown /proc/79/root (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:ksmd 79 root txt unknown /proc/79/exe (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:ksmd 79 root NOFD /proc/79/fd (opendir: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 80 root cwd unknown /proc/80/cwd (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 80 root rtd unknown /proc/80/root (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 80 root txt unknown /proc/80/exe (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 80 root NOFD /proc/80/fd (opendir: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:khugepage 81 root cwd unknown /proc/81/cwd (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:khugepage 81 root rtd unknown /proc/81/root (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:khugepage 81 root txt unknown /proc/81/exe (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:khugepage 81 root NOFD /proc/81/fd (opendir: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:cryptd 82 root cwd unknown /proc/82/cwd (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:cryptd 82 root rtd unknown /proc/82/root (readlink: Permission denied) 2024-03-16T11:08:29.570 INFO:tasks.workunit.client.0.smithi078.stdout:cryptd 82 root txt unknown /proc/82/exe (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:cryptd 82 root NOFD /proc/82/fd (opendir: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kintegrit 83 root cwd unknown /proc/83/cwd (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kintegrit 83 root rtd unknown /proc/83/root (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kintegrit 83 root txt unknown /proc/83/exe (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kintegrit 83 root NOFD /proc/83/fd (opendir: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kblockd 84 root cwd unknown /proc/84/cwd (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kblockd 84 root rtd unknown /proc/84/root (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kblockd 84 root txt unknown /proc/84/exe (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:kblockd 84 root NOFD /proc/84/fd (opendir: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:blkcg_pun 85 root cwd unknown /proc/85/cwd (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:blkcg_pun 85 root rtd unknown /proc/85/root (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:blkcg_pun 85 root txt unknown /proc/85/exe (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:blkcg_pun 85 root NOFD /proc/85/fd (opendir: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:tpm_dev_w 86 root cwd unknown /proc/86/cwd (readlink: Permission denied) 2024-03-16T11:08:29.571 INFO:tasks.workunit.client.0.smithi078.stdout:tpm_dev_w 86 root rtd unknown /proc/86/root (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:tpm_dev_w 86 root txt unknown /proc/86/exe (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:tpm_dev_w 86 root NOFD /proc/86/fd (opendir: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md 87 root cwd unknown /proc/87/cwd (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md 87 root rtd unknown /proc/87/root (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md 87 root txt unknown /proc/87/exe (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md 87 root NOFD /proc/87/fd (opendir: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md_bitmap 88 root cwd unknown /proc/88/cwd (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md_bitmap 88 root rtd unknown /proc/88/root (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md_bitmap 88 root txt unknown /proc/88/exe (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:md_bitmap 88 root NOFD /proc/88/fd (opendir: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:edac-poll 89 root cwd unknown /proc/89/cwd (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:edac-poll 89 root rtd unknown /proc/89/root (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:edac-poll 89 root txt unknown /proc/89/exe (readlink: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:edac-poll 89 root NOFD /proc/89/fd (opendir: Permission denied) 2024-03-16T11:08:29.572 INFO:tasks.workunit.client.0.smithi078.stdout:watchdogd 90 root cwd unknown /proc/90/cwd (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:watchdogd 90 root rtd unknown /proc/90/root (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:watchdogd 90 root txt unknown /proc/90/exe (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:watchdogd 90 root NOFD /proc/90/fd (opendir: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 92 root cwd unknown /proc/92/cwd (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 92 root rtd unknown /proc/92/root (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 92 root txt unknown /proc/92/exe (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 92 root NOFD /proc/92/fd (opendir: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kswapd0 94 root cwd unknown /proc/94/cwd (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kswapd0 94 root rtd unknown /proc/94/root (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kswapd0 94 root txt unknown /proc/94/exe (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kswapd0 94 root NOFD /proc/94/fd (opendir: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kthrotld 100 root cwd unknown /proc/100/cwd (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kthrotld 100 root rtd unknown /proc/100/root (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kthrotld 100 root txt unknown /proc/100/exe (readlink: Permission denied) 2024-03-16T11:08:29.573 INFO:tasks.workunit.client.0.smithi078.stdout:kthrotld 100 root NOFD /proc/100/fd (opendir: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 107 root cwd unknown /proc/107/cwd (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 107 root rtd unknown /proc/107/root (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 107 root txt unknown /proc/107/exe (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 107 root NOFD /proc/107/fd (opendir: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:acpi_ther 108 root cwd unknown /proc/108/cwd (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:acpi_ther 108 root rtd unknown /proc/108/root (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:acpi_ther 108 root txt unknown /proc/108/exe (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:acpi_ther 108 root NOFD /proc/108/fd (opendir: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kmpath_rd 109 root cwd unknown /proc/109/cwd (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kmpath_rd 109 root rtd unknown /proc/109/root (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kmpath_rd 109 root txt unknown /proc/109/exe (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kmpath_rd 109 root NOFD /proc/109/fd (opendir: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kaluad 110 root cwd unknown /proc/110/cwd (readlink: Permission denied) 2024-03-16T11:08:29.574 INFO:tasks.workunit.client.0.smithi078.stdout:kaluad 110 root rtd unknown /proc/110/root (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:kaluad 110 root txt unknown /proc/110/exe (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:kaluad 110 root NOFD /proc/110/fd (opendir: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:mld 112 root cwd unknown /proc/112/cwd (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:mld 112 root rtd unknown /proc/112/root (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:mld 112 root txt unknown /proc/112/exe (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:mld 112 root NOFD /proc/112/fd (opendir: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 113 root cwd unknown /proc/113/cwd (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 113 root rtd unknown /proc/113/root (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 113 root txt unknown /proc/113/exe (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 113 root NOFD /proc/113/fd (opendir: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:ipv6_addr 114 root cwd unknown /proc/114/cwd (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:ipv6_addr 114 root rtd unknown /proc/114/root (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:ipv6_addr 114 root txt unknown /proc/114/exe (readlink: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:ipv6_addr 114 root NOFD /proc/114/fd (opendir: Permission denied) 2024-03-16T11:08:29.575 INFO:tasks.workunit.client.0.smithi078.stdout:kstrp 124 root cwd unknown /proc/124/cwd (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kstrp 124 root rtd unknown /proc/124/root (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kstrp 124 root txt unknown /proc/124/exe (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kstrp 124 root NOFD /proc/124/fd (opendir: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 125 root cwd unknown /proc/125/cwd (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 125 root rtd unknown /proc/125/root (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 125 root txt unknown /proc/125/exe (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 125 root NOFD /proc/125/fd (opendir: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:zswap-shr 131 root cwd unknown /proc/131/cwd (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:zswap-shr 131 root rtd unknown /proc/131/root (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:zswap-shr 131 root txt unknown /proc/131/exe (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:zswap-shr 131 root NOFD /proc/131/fd (opendir: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 132 root cwd unknown /proc/132/cwd (readlink: Permission denied) 2024-03-16T11:08:29.576 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 132 root rtd unknown /proc/132/root (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 132 root txt unknown /proc/132/exe (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 132 root NOFD /proc/132/fd (opendir: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 240 root cwd unknown /proc/240/cwd (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 240 root rtd unknown /proc/240/root (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 240 root txt unknown /proc/240/exe (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 240 root NOFD /proc/240/fd (opendir: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 279 root cwd unknown /proc/279/cwd (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 279 root rtd unknown /proc/279/root (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 279 root txt unknown /proc/279/exe (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 279 root NOFD /proc/279/fd (opendir: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 282 root cwd unknown /proc/282/cwd (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 282 root rtd unknown /proc/282/root (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 282 root txt unknown /proc/282/exe (readlink: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 282 root NOFD /proc/282/fd (opendir: Permission denied) 2024-03-16T11:08:29.577 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 288 root cwd unknown /proc/288/cwd (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 288 root rtd unknown /proc/288/root (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 288 root txt unknown /proc/288/exe (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 288 root NOFD /proc/288/fd (opendir: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 293 root cwd unknown /proc/293/cwd (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 293 root rtd unknown /proc/293/root (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 293 root txt unknown /proc/293/exe (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 293 root NOFD /proc/293/fd (opendir: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 297 root cwd unknown /proc/297/cwd (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 297 root rtd unknown /proc/297/root (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 297 root txt unknown /proc/297/exe (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 297 root NOFD /proc/297/fd (opendir: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 300 root cwd unknown /proc/300/cwd (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 300 root rtd unknown /proc/300/root (readlink: Permission denied) 2024-03-16T11:08:29.578 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 300 root txt unknown /proc/300/exe (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 300 root NOFD /proc/300/fd (opendir: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 330 root cwd unknown /proc/330/cwd (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 330 root rtd unknown /proc/330/root (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 330 root txt unknown /proc/330/exe (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 330 root NOFD /proc/330/fd (opendir: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:ata_sff 431 root cwd unknown /proc/431/cwd (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:ata_sff 431 root rtd unknown /proc/431/root (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:ata_sff 431 root txt unknown /proc/431/exe (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:ata_sff 431 root NOFD /proc/431/fd (opendir: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-wq 437 root cwd unknown /proc/437/cwd (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-wq 437 root rtd unknown /proc/437/root (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-wq 437 root txt unknown /proc/437/exe (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-wq 437 root NOFD /proc/437/fd (opendir: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-rese 439 root cwd unknown /proc/439/cwd (readlink: Permission denied) 2024-03-16T11:08:29.579 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-rese 439 root rtd unknown /proc/439/root (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-rese 439 root txt unknown /proc/439/exe (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-rese 439 root NOFD /proc/439/fd (opendir: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-dele 441 root cwd unknown /proc/441/cwd (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-dele 441 root rtd unknown /proc/441/root (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-dele 441 root txt unknown /proc/441/exe (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-dele 441 root NOFD /proc/441/fd (opendir: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-auth 442 root cwd unknown /proc/442/cwd (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-auth 442 root rtd unknown /proc/442/root (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-auth 442 root txt unknown /proc/442/exe (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:nvme-auth 442 root NOFD /proc/442/fd (opendir: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:ixgbe 445 root cwd unknown /proc/445/cwd (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:ixgbe 445 root rtd unknown /proc/445/root (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:ixgbe 445 root txt unknown /proc/445/exe (readlink: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:ixgbe 445 root NOFD /proc/445/fd (opendir: Permission denied) 2024-03-16T11:08:29.580 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_0 448 root cwd unknown /proc/448/cwd (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_0 448 root rtd unknown /proc/448/root (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_0 448 root txt unknown /proc/448/exe (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_0 448 root NOFD /proc/448/fd (opendir: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 449 root cwd unknown /proc/449/cwd (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 449 root rtd unknown /proc/449/root (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 449 root txt unknown /proc/449/exe (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 449 root NOFD /proc/449/fd (opendir: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_1 450 root cwd unknown /proc/450/cwd (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_1 450 root rtd unknown /proc/450/root (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_1 450 root txt unknown /proc/450/exe (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_1 450 root NOFD /proc/450/fd (opendir: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 451 root cwd unknown /proc/451/cwd (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 451 root rtd unknown /proc/451/root (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 451 root txt unknown /proc/451/exe (readlink: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 451 root NOFD /proc/451/fd (opendir: Permission denied) 2024-03-16T11:08:29.581 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_2 452 root cwd unknown /proc/452/cwd (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_2 452 root rtd unknown /proc/452/root (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_2 452 root txt unknown /proc/452/exe (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_2 452 root NOFD /proc/452/fd (opendir: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 453 root cwd unknown /proc/453/cwd (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 453 root rtd unknown /proc/453/root (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 453 root txt unknown /proc/453/exe (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 453 root NOFD /proc/453/fd (opendir: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_3 454 root cwd unknown /proc/454/cwd (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_3 454 root rtd unknown /proc/454/root (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_3 454 root txt unknown /proc/454/exe (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_3 454 root NOFD /proc/454/fd (opendir: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 455 root cwd unknown /proc/455/cwd (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 455 root rtd unknown /proc/455/root (readlink: Permission denied) 2024-03-16T11:08:29.582 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 455 root txt unknown /proc/455/exe (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 455 root NOFD /proc/455/fd (opendir: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_4 459 root cwd unknown /proc/459/cwd (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_4 459 root rtd unknown /proc/459/root (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_4 459 root txt unknown /proc/459/exe (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_4 459 root NOFD /proc/459/fd (opendir: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 460 root cwd unknown /proc/460/cwd (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 460 root rtd unknown /proc/460/root (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 460 root txt unknown /proc/460/exe (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 460 root NOFD /proc/460/fd (opendir: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_5 461 root cwd unknown /proc/461/cwd (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_5 461 root rtd unknown /proc/461/root (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_5 461 root txt unknown /proc/461/exe (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_5 461 root NOFD /proc/461/fd (opendir: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 462 root cwd unknown /proc/462/cwd (readlink: Permission denied) 2024-03-16T11:08:29.583 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 462 root rtd unknown /proc/462/root (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 462 root txt unknown /proc/462/exe (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 462 root NOFD /proc/462/fd (opendir: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_6 463 root cwd unknown /proc/463/cwd (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_6 463 root rtd unknown /proc/463/root (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_6 463 root txt unknown /proc/463/exe (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_6 463 root NOFD /proc/463/fd (opendir: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 464 root cwd unknown /proc/464/cwd (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 464 root rtd unknown /proc/464/root (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 464 root txt unknown /proc/464/exe (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 464 root NOFD /proc/464/fd (opendir: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_7 465 root cwd unknown /proc/465/cwd (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_7 465 root rtd unknown /proc/465/root (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_7 465 root txt unknown /proc/465/exe (readlink: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_7 465 root NOFD /proc/465/fd (opendir: Permission denied) 2024-03-16T11:08:29.584 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 466 root cwd unknown /proc/466/cwd (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 466 root rtd unknown /proc/466/root (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 466 root txt unknown /proc/466/exe (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 466 root NOFD /proc/466/fd (opendir: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_8 467 root cwd unknown /proc/467/cwd (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_8 467 root rtd unknown /proc/467/root (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_8 467 root txt unknown /proc/467/exe (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_8 467 root NOFD /proc/467/fd (opendir: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 468 root cwd unknown /proc/468/cwd (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 468 root rtd unknown /proc/468/root (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 468 root txt unknown /proc/468/exe (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 468 root NOFD /proc/468/fd (opendir: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_9 469 root cwd unknown /proc/469/cwd (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_9 469 root rtd unknown /proc/469/root (readlink: Permission denied) 2024-03-16T11:08:29.585 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_9 469 root txt unknown /proc/469/exe (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_eh_9 469 root NOFD /proc/469/fd (opendir: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 470 root cwd unknown /proc/470/cwd (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 470 root rtd unknown /proc/470/root (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 470 root txt unknown /proc/470/exe (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:scsi_tmf_ 470 root NOFD /proc/470/fd (opendir: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 471 root cwd unknown /proc/471/cwd (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 471 root rtd unknown /proc/471/root (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 471 root txt unknown /proc/471/exe (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 471 root NOFD /proc/471/fd (opendir: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 472 root cwd unknown /proc/472/cwd (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 472 root rtd unknown /proc/472/root (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 472 root txt unknown /proc/472/exe (readlink: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 472 root NOFD /proc/472/fd (opendir: Permission denied) 2024-03-16T11:08:29.586 INFO:tasks.workunit.client.0.smithi078.stdout:jbd2/sda1 501 root cwd unknown /proc/501/cwd (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:jbd2/sda1 501 root rtd unknown /proc/501/root (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:jbd2/sda1 501 root txt unknown /proc/501/exe (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:jbd2/sda1 501 root NOFD /proc/501/fd (opendir: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:ext4-rsv- 502 root cwd unknown /proc/502/cwd (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:ext4-rsv- 502 root rtd unknown /proc/502/root (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:ext4-rsv- 502 root txt unknown /proc/502/exe (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:ext4-rsv- 502 root NOFD /proc/502/fd (opendir: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-j 578 root cwd unknown /proc/578/cwd (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-j 578 root rtd unknown /proc/578/root (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-j 578 root txt unknown /proc/578/exe (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-j 578 root NOFD /proc/578/fd (opendir: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:ext4lazyi 584 root cwd unknown /proc/584/cwd (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:ext4lazyi 584 root rtd unknown /proc/584/root (readlink: Permission denied) 2024-03-16T11:08:29.587 INFO:tasks.workunit.client.0.smithi078.stdout:ext4lazyi 584 root txt unknown /proc/584/exe (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:ext4lazyi 584 root NOFD /proc/584/fd (opendir: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-u 595 root cwd unknown /proc/595/cwd (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-u 595 root rtd unknown /proc/595/root (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-u 595 root txt unknown /proc/595/exe (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-u 595 root NOFD /proc/595/fd (opendir: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:ipmi-msgh 637 root cwd unknown /proc/637/cwd (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:ipmi-msgh 637 root rtd unknown /proc/637/root (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:ipmi-msgh 637 root txt unknown /proc/637/exe (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:ipmi-msgh 637 root NOFD /proc/637/fd (opendir: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:kipmi0 646 root cwd unknown /proc/646/cwd (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:kipmi0 646 root rtd unknown /proc/646/root (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:kipmi0 646 root txt unknown /proc/646/exe (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:kipmi0 646 root NOFD /proc/646/fd (opendir: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:rpcbind 669 rpc cwd unknown /proc/669/cwd (readlink: Permission denied) 2024-03-16T11:08:29.588 INFO:tasks.workunit.client.0.smithi078.stdout:rpcbind 669 rpc rtd unknown /proc/669/root (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:rpcbind 669 rpc txt unknown /proc/669/exe (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:rpcbind 669 rpc NOFD /proc/669/fd (opendir: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:rpciod 674 root cwd unknown /proc/674/cwd (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:rpciod 674 root rtd unknown /proc/674/root (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:rpciod 674 root txt unknown /proc/674/exe (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:rpciod 674 root NOFD /proc/674/fd (opendir: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:xprtiod 675 root cwd unknown /proc/675/cwd (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:xprtiod 675 root rtd unknown /proc/675/root (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:xprtiod 675 root txt unknown /proc/675/exe (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:xprtiod 675 root NOFD /proc/675/fd (opendir: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 root cwd unknown /proc/676/cwd (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 root rtd unknown /proc/676/root (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 root txt unknown /proc/676/exe (readlink: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 root NOFD /proc/676/fd (opendir: Permission denied) 2024-03-16T11:08:29.589 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 677 auditd root cwd unknown /proc/676/task/677/cwd (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 677 auditd root rtd unknown /proc/676/task/677/root (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 677 auditd root txt unknown /proc/676/task/677/exe (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 677 auditd root NOFD /proc/676/task/677/fd (opendir: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 679 auditd root cwd unknown /proc/676/task/679/cwd (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 679 auditd root rtd unknown /proc/676/task/679/root (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 679 auditd root txt unknown /proc/676/task/679/exe (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:auditd 676 679 auditd root NOFD /proc/676/task/679/fd (opendir: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:sedispatc 678 root cwd unknown /proc/678/cwd (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:sedispatc 678 root rtd unknown /proc/678/root (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:sedispatc 678 root txt unknown /proc/678/exe (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:sedispatc 678 root NOFD /proc/678/fd (opendir: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 715 dbus cwd unknown /proc/715/cwd (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 715 dbus rtd unknown /proc/715/root (readlink: Permission denied) 2024-03-16T11:08:29.590 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 715 dbus txt unknown /proc/715/exe (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 715 dbus NOFD /proc/715/fd (opendir: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 716 dbus cwd unknown /proc/716/cwd (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 716 dbus rtd unknown /proc/716/root (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 716 dbus txt unknown /proc/716/exe (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:dbus-brok 716 dbus NOFD /proc/716/fd (opendir: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 root cwd unknown /proc/717/cwd (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 root rtd unknown /proc/717/root (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 root txt unknown /proc/717/exe (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 root NOFD /proc/717/fd (opendir: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 737 gmain root cwd unknown /proc/717/task/737/cwd (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 737 gmain root rtd unknown /proc/717/task/737/root (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 737 gmain root txt unknown /proc/717/task/737/exe (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 737 gmain root NOFD /proc/717/task/737/fd (opendir: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 738 gdbus root cwd unknown /proc/717/task/738/cwd (readlink: Permission denied) 2024-03-16T11:08:29.591 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 738 gdbus root rtd unknown /proc/717/task/738/root (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 738 gdbus root txt unknown /proc/717/task/738/exe (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:NetworkMa 717 738 gdbus root NOFD /proc/717/task/738/fd (opendir: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 root cwd unknown /proc/720/cwd (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 root rtd unknown /proc/720/root (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 root txt unknown /proc/720/exe (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 root NOFD /proc/720/fd (opendir: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 727 gmain root cwd unknown /proc/720/task/727/cwd (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 727 gmain root rtd unknown /proc/720/task/727/root (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 727 gmain root txt unknown /proc/720/task/727/exe (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:irqbalanc 720 727 gmain root NOFD /proc/720/task/727/fd (opendir: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:lsmd 721 libstoragemgmt cwd unknown /proc/721/cwd (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:lsmd 721 libstoragemgmt rtd unknown /proc/721/root (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:lsmd 721 libstoragemgmt txt unknown /proc/721/exe (readlink: Permission denied) 2024-03-16T11:08:29.592 INFO:tasks.workunit.client.0.smithi078.stdout:lsmd 721 libstoragemgmt NOFD /proc/721/fd (opendir: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:mcelog 722 root cwd unknown /proc/722/cwd (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:mcelog 722 root rtd unknown /proc/722/root (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:mcelog 722 root txt unknown /proc/722/exe (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:mcelog 722 root NOFD /proc/722/fd (opendir: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 polkitd cwd unknown /proc/723/cwd (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 polkitd rtd unknown /proc/723/root (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 polkitd txt unknown /proc/723/exe (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 polkitd NOFD /proc/723/fd (opendir: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 735 gmain polkitd cwd unknown /proc/723/task/735/cwd (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 735 gmain polkitd rtd unknown /proc/723/task/735/root (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 735 gmain polkitd txt unknown /proc/723/task/735/exe (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 735 gmain polkitd NOFD /proc/723/task/735/fd (opendir: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 736 gdbus polkitd cwd unknown /proc/723/task/736/cwd (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 736 gdbus polkitd rtd unknown /proc/723/task/736/root (readlink: Permission denied) 2024-03-16T11:08:29.593 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 736 gdbus polkitd txt unknown /proc/723/task/736/exe (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 736 gdbus polkitd NOFD /proc/723/task/736/fd (opendir: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 739 JS\x20Hel polkitd cwd unknown /proc/723/task/739/cwd (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 739 JS\x20Hel polkitd rtd unknown /proc/723/task/739/root (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 739 JS\x20Hel polkitd txt unknown /proc/723/task/739/exe (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 739 JS\x20Hel polkitd NOFD /proc/723/task/739/fd (opendir: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 740 JS\x20Hel polkitd cwd unknown /proc/723/task/740/cwd (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 740 JS\x20Hel polkitd rtd unknown /proc/723/task/740/root (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 740 JS\x20Hel polkitd txt unknown /proc/723/task/740/exe (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 740 JS\x20Hel polkitd NOFD /proc/723/task/740/fd (opendir: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 741 JS\x20Hel polkitd cwd unknown /proc/723/task/741/cwd (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 741 JS\x20Hel polkitd rtd unknown /proc/723/task/741/root (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 741 JS\x20Hel polkitd txt unknown /proc/723/task/741/exe (readlink: Permission denied) 2024-03-16T11:08:29.594 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 741 JS\x20Hel polkitd NOFD /proc/723/task/741/fd (opendir: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 742 JS\x20Hel polkitd cwd unknown /proc/723/task/742/cwd (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 742 JS\x20Hel polkitd rtd unknown /proc/723/task/742/root (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 742 JS\x20Hel polkitd txt unknown /proc/723/task/742/exe (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 742 JS\x20Hel polkitd NOFD /proc/723/task/742/fd (opendir: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 743 JS\x20Hel polkitd cwd unknown /proc/723/task/743/cwd (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 743 JS\x20Hel polkitd rtd unknown /proc/723/task/743/root (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 743 JS\x20Hel polkitd txt unknown /proc/723/task/743/exe (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 743 JS\x20Hel polkitd NOFD /proc/723/task/743/fd (opendir: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 744 JS\x20Hel polkitd cwd unknown /proc/723/task/744/cwd (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 744 JS\x20Hel polkitd rtd unknown /proc/723/task/744/root (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 744 JS\x20Hel polkitd txt unknown /proc/723/task/744/exe (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 744 JS\x20Hel polkitd NOFD /proc/723/task/744/fd (opendir: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 745 JS\x20Hel polkitd cwd unknown /proc/723/task/745/cwd (readlink: Permission denied) 2024-03-16T11:08:29.595 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 745 JS\x20Hel polkitd rtd unknown /proc/723/task/745/root (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 745 JS\x20Hel polkitd txt unknown /proc/723/task/745/exe (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 745 JS\x20Hel polkitd NOFD /proc/723/task/745/fd (opendir: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 746 JS\x20Hel polkitd cwd unknown /proc/723/task/746/cwd (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 746 JS\x20Hel polkitd rtd unknown /proc/723/task/746/root (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 746 JS\x20Hel polkitd txt unknown /proc/723/task/746/exe (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 746 JS\x20Hel polkitd NOFD /proc/723/task/746/fd (opendir: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 749 polkitd polkitd cwd unknown /proc/723/task/749/cwd (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 749 polkitd polkitd rtd unknown /proc/723/task/749/root (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 749 polkitd polkitd txt unknown /proc/723/task/749/exe (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:polkitd 723 749 polkitd polkitd NOFD /proc/723/task/749/fd (opendir: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-l 726 root cwd unknown /proc/726/cwd (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-l 726 root rtd unknown /proc/726/root (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-l 726 root txt unknown /proc/726/exe (readlink: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:systemd-l 726 root NOFD /proc/726/fd (opendir: Permission denied) 2024-03-16T11:08:29.596 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 779 root cwd unknown /proc/779/cwd (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 779 root rtd unknown /proc/779/root (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 779 root txt unknown /proc/779/exe (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 779 root NOFD /proc/779/fd (opendir: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 root cwd unknown /proc/780/cwd (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 root rtd unknown /proc/780/root (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 root txt unknown /proc/780/exe (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 root NOFD /proc/780/fd (opendir: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 865 tuned root cwd unknown /proc/780/task/865/cwd (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 865 tuned root rtd unknown /proc/780/task/865/root (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 865 tuned root txt unknown /proc/780/task/865/exe (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:tuned 780 865 tuned root NOFD /proc/780/task/865/fd (opendir: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 root cwd unknown /proc/784/cwd (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 root rtd unknown /proc/784/root (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 root txt unknown /proc/784/exe (readlink: Permission denied) 2024-03-16T11:08:29.597 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 root NOFD /proc/784/fd (opendir: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 785 gssproxy root cwd unknown /proc/784/task/785/cwd (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 785 gssproxy root rtd unknown /proc/784/task/785/root (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 785 gssproxy root txt unknown /proc/784/task/785/exe (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 785 gssproxy root NOFD /proc/784/task/785/fd (opendir: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 786 gssproxy root cwd unknown /proc/784/task/786/cwd (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 786 gssproxy root rtd unknown /proc/784/task/786/root (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 786 gssproxy root txt unknown /proc/784/task/786/exe (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 786 gssproxy root NOFD /proc/784/task/786/fd (opendir: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 787 gssproxy root cwd unknown /proc/784/task/787/cwd (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 787 gssproxy root rtd unknown /proc/784/task/787/root (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 787 gssproxy root txt unknown /proc/784/task/787/exe (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 787 gssproxy root NOFD /proc/784/task/787/fd (opendir: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 788 gssproxy root cwd unknown /proc/784/task/788/cwd (readlink: Permission denied) 2024-03-16T11:08:29.598 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 788 gssproxy root rtd unknown /proc/784/task/788/root (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 788 gssproxy root txt unknown /proc/784/task/788/exe (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 788 gssproxy root NOFD /proc/784/task/788/fd (opendir: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 789 gssproxy root cwd unknown /proc/784/task/789/cwd (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 789 gssproxy root rtd unknown /proc/784/task/789/root (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 789 gssproxy root txt unknown /proc/784/task/789/exe (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:gssproxy 784 789 gssproxy root NOFD /proc/784/task/789/fd (opendir: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:atd 791 root cwd unknown /proc/791/cwd (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:atd 791 root rtd unknown /proc/791/root (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:atd 791 root txt unknown /proc/791/exe (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:atd 791 root NOFD /proc/791/fd (opendir: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:crond 792 root cwd unknown /proc/792/cwd (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:crond 792 root rtd unknown /proc/792/root (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:crond 792 root txt unknown /proc/792/exe (readlink: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:crond 792 root NOFD /proc/792/fd (opendir: Permission denied) 2024-03-16T11:08:29.599 INFO:tasks.workunit.client.0.smithi078.stdout:nrpe 886 nrpe cwd unknown /proc/886/cwd (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:nrpe 886 nrpe rtd unknown /proc/886/root (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:nrpe 886 nrpe txt unknown /proc/886/exe (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:nrpe 886 nrpe NOFD /proc/886/fd (opendir: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1365 root cwd unknown /proc/1365/cwd (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1365 root rtd unknown /proc/1365/root (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1365 root txt unknown /proc/1365/exe (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1365 root NOFD /proc/1365/fd (opendir: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1369 root cwd unknown /proc/1369/cwd (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1369 root rtd unknown /proc/1369/root (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1369 root txt unknown /proc/1369/exe (readlink: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:agetty 1369 root NOFD /proc/1369/fd (opendir: Permission denied) 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu cwd DIR 8,1 4096 2 / 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-03-16T11:08:29.600 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu DEL REG 8,1 1143 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu DEL REG 8,1 10007 /usr/lib64/libcrypto.so.3.0.7 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu DEL REG 8,1 1697 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-03-16T11:08:29.601 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu DEL REG 8,1 10126 /usr/lib64/libblkid.so.1.1.0 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu DEL REG 8,1 9806 /usr/lib64/libacl.so.1.1.2301 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu DEL REG 8,1 10156 /usr/lib64/libmount.so.1.1.0 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-03-16T11:08:29.602 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 1u unix 0x0000000000000000 0t0 19435 type=STREAM (CONNECTED) 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 2u unix 0x0000000000000000 0t0 19435 type=STREAM (CONNECTED) 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 3u unix 0x0000000000000000 0t0 21445 type=DGRAM (CONNECTED) 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 4u a_inode 0,14 0 1085 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 5u a_inode 0,14 0 1085 [signalfd] 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 6r a_inode 0,14 0 1085 inotify 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 7r DIR 0,25 0 3393 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 8u a_inode 0,14 0 1085 [timerfd] 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 9u a_inode 0,14 0 1085 [eventpoll:10,13] 2024-03-16T11:08:29.603 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 10r REG 0,20 0 24961 /proc/4771/mountinfo 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 11r a_inode 0,14 0 1085 inotify 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 12u unix 0x0000000000000000 0t0 21467 /run/user/1000/bus type=STREAM (LISTEN) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 13r a_inode 0,14 0 1085 inotify 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 14r REG 0,20 0 4026532076 /proc/swaps 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 15u netlink 0t0 21449 KOBJECT_UEVENT 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 16u unix 0x0000000000000000 0t0 21454 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 17u unix 0x0000000000000000 0t0 21455 type=DGRAM (CONNECTED) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 18u unix 0x0000000000000000 0t0 21456 type=DGRAM (CONNECTED) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 19u unix 0x0000000000000000 0t0 21457 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 20u unix 0x0000000000000000 0t0 21459 type=STREAM (CONNECTED) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 4771 ubuntu 21u a_inode 0,14 0 1085 [timerfd] 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 4773 ubuntu cwd unknown /proc/4773/cwd (readlink: Permission denied) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 4773 ubuntu rtd unknown /proc/4773/root (readlink: Permission denied) 2024-03-16T11:08:29.604 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 4773 ubuntu txt unknown /proc/4773/exe (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 4773 ubuntu NOFD /proc/4773/fd (opendir: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 root cwd unknown /proc/5108/cwd (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 root rtd unknown /proc/5108/root (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 root txt unknown /proc/5108/exe (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 root NOFD /proc/5108/fd (opendir: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5110 in:imjour root cwd unknown /proc/5108/task/5110/cwd (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5110 in:imjour root rtd unknown /proc/5108/task/5110/root (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5110 in:imjour root txt unknown /proc/5108/task/5110/exe (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5110 in:imjour root NOFD /proc/5108/task/5110/fd (opendir: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5111 rs:main root cwd unknown /proc/5108/task/5111/cwd (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5111 rs:main root rtd unknown /proc/5108/task/5111/root (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5111 rs:main root txt unknown /proc/5108/task/5111/exe (readlink: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:rsyslogd 5108 5111 rs:main root NOFD /proc/5108/task/5111/fd (opendir: Permission denied) 2024-03-16T11:08:29.605 INFO:tasks.workunit.client.0.smithi078.stdout:tls-strp 15812 root cwd unknown /proc/15812/cwd (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:tls-strp 15812 root rtd unknown /proc/15812/root (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:tls-strp 15812 root txt unknown /proc/15812/exe (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:tls-strp 15812 root NOFD /proc/15812/fd (opendir: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:smartd 15860 root cwd unknown /proc/15860/cwd (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:smartd 15860 root rtd unknown /proc/15860/root (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:smartd 15860 root txt unknown /proc/15860/exe (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:smartd 15860 root NOFD /proc/15860/fd (opendir: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21211 root cwd unknown /proc/21211/cwd (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21211 root rtd unknown /proc/21211/root (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21211 root txt unknown /proc/21211/exe (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21211 root NOFD /proc/21211/fd (opendir: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21240 root cwd unknown /proc/21240/cwd (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21240 root rtd unknown /proc/21240/root (readlink: Permission denied) 2024-03-16T11:08:29.606 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21240 root txt unknown /proc/21240/exe (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21240 root NOFD /proc/21240/fd (opendir: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21269 root cwd unknown /proc/21269/cwd (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21269 root rtd unknown /proc/21269/root (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21269 root txt unknown /proc/21269/exe (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21269 root NOFD /proc/21269/fd (opendir: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21298 root cwd unknown /proc/21298/cwd (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21298 root rtd unknown /proc/21298/root (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21298 root txt unknown /proc/21298/exe (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21298 root NOFD /proc/21298/fd (opendir: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21327 root cwd unknown /proc/21327/cwd (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21327 root rtd unknown /proc/21327/root (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21327 root txt unknown /proc/21327/exe (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:kdmflush/ 21327 root NOFD /proc/21327/fd (opendir: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:xfsalloc 21491 root cwd unknown /proc/21491/cwd (readlink: Permission denied) 2024-03-16T11:08:29.607 INFO:tasks.workunit.client.0.smithi078.stdout:xfsalloc 21491 root rtd unknown /proc/21491/root (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfsalloc 21491 root txt unknown /proc/21491/exe (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfsalloc 21491 root NOFD /proc/21491/fd (opendir: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs_mru_c 21492 root cwd unknown /proc/21492/cwd (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs_mru_c 21492 root rtd unknown /proc/21492/root (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs_mru_c 21492 root txt unknown /proc/21492/exe (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs_mru_c 21492 root NOFD /proc/21492/fd (opendir: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-buf/d 21493 root cwd unknown /proc/21493/cwd (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-buf/d 21493 root rtd unknown /proc/21493/root (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-buf/d 21493 root txt unknown /proc/21493/exe (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-buf/d 21493 root NOFD /proc/21493/fd (opendir: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-conv/ 21494 root cwd unknown /proc/21494/cwd (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-conv/ 21494 root rtd unknown /proc/21494/root (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-conv/ 21494 root txt unknown /proc/21494/exe (readlink: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-conv/ 21494 root NOFD /proc/21494/fd (opendir: Permission denied) 2024-03-16T11:08:29.608 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-recla 21495 root cwd unknown /proc/21495/cwd (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-recla 21495 root rtd unknown /proc/21495/root (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-recla 21495 root txt unknown /proc/21495/exe (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-recla 21495 root NOFD /proc/21495/fd (opendir: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-block 21496 root cwd unknown /proc/21496/cwd (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-block 21496 root rtd unknown /proc/21496/root (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-block 21496 root txt unknown /proc/21496/exe (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-block 21496 root NOFD /proc/21496/fd (opendir: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-inode 21497 root cwd unknown /proc/21497/cwd (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-inode 21497 root rtd unknown /proc/21497/root (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-inode 21497 root txt unknown /proc/21497/exe (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-inode 21497 root NOFD /proc/21497/fd (opendir: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-log/d 21498 root cwd unknown /proc/21498/cwd (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-log/d 21498 root rtd unknown /proc/21498/root (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-log/d 21498 root txt unknown /proc/21498/exe (readlink: Permission denied) 2024-03-16T11:08:29.609 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-log/d 21498 root NOFD /proc/21498/fd (opendir: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-cil/d 21499 root cwd unknown /proc/21499/cwd (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-cil/d 21499 root rtd unknown /proc/21499/root (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-cil/d 21499 root txt unknown /proc/21499/exe (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfs-cil/d 21499 root NOFD /proc/21499/fd (opendir: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfsaild/d 21500 root cwd unknown /proc/21500/cwd (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfsaild/d 21500 root rtd unknown /proc/21500/root (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfsaild/d 21500 root txt unknown /proc/21500/exe (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:xfsaild/d 21500 root NOFD /proc/21500/fd (opendir: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23434 root cwd unknown /proc/23434/cwd (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23434 root rtd unknown /proc/23434/root (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23434 root txt unknown /proc/23434/exe (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23434 root NOFD /proc/23434/fd (opendir: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23437 ubuntu cwd unknown /proc/23437/cwd (readlink: Permission denied) 2024-03-16T11:08:29.610 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23437 ubuntu rtd unknown /proc/23437/root (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23437 ubuntu txt unknown /proc/23437/exe (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 23437 ubuntu NOFD /proc/23437/fd (opendir: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:chronyd 23497 chrony cwd unknown /proc/23497/cwd (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:chronyd 23497 chrony rtd unknown /proc/23497/root (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:chronyd 23497 chrony txt unknown /proc/23497/exe (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:chronyd 23497 chrony NOFD /proc/23497/fd (opendir: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-cras 23780 ceph cwd unknown /proc/23780/cwd (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-cras 23780 ceph rtd unknown /proc/23780/root (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-cras 23780 ceph txt unknown /proc/23780/exe (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-cras 23780 ceph NOFD /proc/23780/fd (opendir: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 24480 root cwd unknown /proc/24480/cwd (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 24480 root rtd unknown /proc/24480/root (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 24480 root txt unknown /proc/24480/exe (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/u 24480 root NOFD /proc/24480/fd (opendir: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25598 root cwd unknown /proc/25598/cwd (readlink: Permission denied) 2024-03-16T11:08:29.611 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25598 root rtd unknown /proc/25598/root (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25598 root txt unknown /proc/25598/exe (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25598 root NOFD /proc/25598/fd (opendir: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25599 root cwd unknown /proc/25599/cwd (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25599 root rtd unknown /proc/25599/root (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25599 root txt unknown /proc/25599/exe (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 25599 root NOFD /proc/25599/fd (opendir: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 26259 root cwd unknown /proc/26259/cwd (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 26259 root rtd unknown /proc/26259/root (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 26259 root txt unknown /proc/26259/exe (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 26259 root NOFD /proc/26259/fd (opendir: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26801 root cwd unknown /proc/26801/cwd (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26801 root rtd unknown /proc/26801/root (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26801 root txt unknown /proc/26801/exe (readlink: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26801 root NOFD /proc/26801/fd (opendir: Permission denied) 2024-03-16T11:08:29.612 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26802 root cwd unknown /proc/26802/cwd (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26802 root rtd unknown /proc/26802/root (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26802 root txt unknown /proc/26802/exe (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 26802 root NOFD /proc/26802/fd (opendir: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 31061 root cwd unknown /proc/31061/cwd (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 31061 root rtd unknown /proc/31061/root (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 31061 root txt unknown /proc/31061/exe (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 31061 root NOFD /proc/31061/fd (opendir: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31190 root cwd unknown /proc/31190/cwd (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31190 root rtd unknown /proc/31190/root (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31190 root txt unknown /proc/31190/exe (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31190 root NOFD /proc/31190/fd (opendir: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31207 root cwd unknown /proc/31207/cwd (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31207 root rtd unknown /proc/31207/root (readlink: Permission denied) 2024-03-16T11:08:29.613 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31207 root txt unknown /proc/31207/exe (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31207 root NOFD /proc/31207/fd (opendir: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31208 root cwd unknown /proc/31208/cwd (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31208 root rtd unknown /proc/31208/root (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31208 root txt unknown /proc/31208/exe (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 31208 root NOFD /proc/31208/fd (opendir: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31225 root cwd unknown /proc/31225/cwd (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31225 root rtd unknown /proc/31225/root (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31225 root txt unknown /proc/31225/exe (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 31225 root NOFD /proc/31225/fd (opendir: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32177 root cwd unknown /proc/32177/cwd (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32177 root rtd unknown /proc/32177/root (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32177 root txt unknown /proc/32177/exe (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 32177 root NOFD /proc/32177/fd (opendir: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32231 root cwd unknown /proc/32231/cwd (readlink: Permission denied) 2024-03-16T11:08:29.614 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32231 root rtd unknown /proc/32231/root (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32231 root txt unknown /proc/32231/exe (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32231 root NOFD /proc/32231/fd (opendir: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32233 root cwd unknown /proc/32233/cwd (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32233 root rtd unknown /proc/32233/root (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32233 root txt unknown /proc/32233/exe (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32233 root NOFD /proc/32233/fd (opendir: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 ceph cwd unknown /proc/32235/cwd (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 ceph rtd unknown /proc/32235/root (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 ceph txt unknown /proc/32235/exe (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 ceph NOFD /proc/32235/fd (opendir: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32236 log ceph cwd unknown /proc/32235/task/32236/cwd (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32236 log ceph rtd unknown /proc/32235/task/32236/root (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32236 log ceph txt unknown /proc/32235/task/32236/exe (readlink: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32236 log ceph NOFD /proc/32235/task/32236/fd (opendir: Permission denied) 2024-03-16T11:08:29.615 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32237 service ceph cwd unknown /proc/32235/task/32237/cwd (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32237 service ceph rtd unknown /proc/32235/task/32237/root (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32237 service ceph txt unknown /proc/32235/task/32237/exe (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32237 service ceph NOFD /proc/32235/task/32237/fd (opendir: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32238 admin_soc ceph cwd unknown /proc/32235/task/32238/cwd (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32238 admin_soc ceph rtd unknown /proc/32235/task/32238/root (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32238 admin_soc ceph txt unknown /proc/32235/task/32238/exe (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32238 admin_soc ceph NOFD /proc/32235/task/32238/fd (opendir: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32239 signal_ha ceph cwd unknown /proc/32235/task/32239/cwd (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32239 signal_ha ceph rtd unknown /proc/32235/task/32239/root (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32239 signal_ha ceph txt unknown /proc/32235/task/32239/exe (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32239 signal_ha ceph NOFD /proc/32235/task/32239/fd (opendir: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32240 rocksdb:l ceph cwd unknown /proc/32235/task/32240/cwd (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32240 rocksdb:l ceph rtd unknown /proc/32235/task/32240/root (readlink: Permission denied) 2024-03-16T11:08:29.616 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32240 rocksdb:l ceph txt unknown /proc/32235/task/32240/exe (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32240 rocksdb:l ceph NOFD /proc/32235/task/32240/fd (opendir: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32241 rocksdb:h ceph cwd unknown /proc/32235/task/32241/cwd (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32241 rocksdb:h ceph rtd unknown /proc/32235/task/32241/root (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32241 rocksdb:h ceph txt unknown /proc/32235/task/32241/exe (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32241 rocksdb:h ceph NOFD /proc/32235/task/32241/fd (opendir: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32257 ceph-mon ceph cwd unknown /proc/32235/task/32257/cwd (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32257 ceph-mon ceph rtd unknown /proc/32235/task/32257/root (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32257 ceph-mon ceph txt unknown /proc/32235/task/32257/exe (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32257 ceph-mon ceph NOFD /proc/32235/task/32257/fd (opendir: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32258 fn_monsto ceph cwd unknown /proc/32235/task/32258/cwd (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32258 fn_monsto ceph rtd unknown /proc/32235/task/32258/root (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32258 fn_monsto ceph txt unknown /proc/32235/task/32258/exe (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32258 fn_monsto ceph NOFD /proc/32235/task/32258/fd (opendir: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32259 msgr-work ceph cwd unknown /proc/32235/task/32259/cwd (readlink: Permission denied) 2024-03-16T11:08:29.617 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32259 msgr-work ceph rtd unknown /proc/32235/task/32259/root (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32259 msgr-work ceph txt unknown /proc/32235/task/32259/exe (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32259 msgr-work ceph NOFD /proc/32235/task/32259/fd (opendir: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32260 msgr-work ceph cwd unknown /proc/32235/task/32260/cwd (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32260 msgr-work ceph rtd unknown /proc/32235/task/32260/root (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32260 msgr-work ceph txt unknown /proc/32235/task/32260/exe (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32260 msgr-work ceph NOFD /proc/32235/task/32260/fd (opendir: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32261 msgr-work ceph cwd unknown /proc/32235/task/32261/cwd (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32261 msgr-work ceph rtd unknown /proc/32235/task/32261/root (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32261 msgr-work ceph txt unknown /proc/32235/task/32261/exe (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32261 msgr-work ceph NOFD /proc/32235/task/32261/fd (opendir: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32262 OpHistory ceph cwd unknown /proc/32235/task/32262/cwd (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32262 OpHistory ceph rtd unknown /proc/32235/task/32262/root (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32262 OpHistory ceph txt unknown /proc/32235/task/32262/exe (readlink: Permission denied) 2024-03-16T11:08:29.618 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32262 OpHistory ceph NOFD /proc/32235/task/32262/fd (opendir: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32263 fin ceph cwd unknown /proc/32235/task/32263/cwd (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32263 fin ceph rtd unknown /proc/32235/task/32263/root (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32263 fin ceph txt unknown /proc/32235/task/32263/exe (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32263 fin ceph NOFD /proc/32235/task/32263/fd (opendir: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32264 safe_time ceph cwd unknown /proc/32235/task/32264/cwd (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32264 safe_time ceph rtd unknown /proc/32235/task/32264/root (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32264 safe_time ceph txt unknown /proc/32235/task/32264/exe (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32264 safe_time ceph NOFD /proc/32235/task/32264/fd (opendir: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32265 cpu_tp ceph cwd unknown /proc/32235/task/32265/cwd (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32265 cpu_tp ceph rtd unknown /proc/32235/task/32265/root (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32265 cpu_tp ceph txt unknown /proc/32235/task/32265/exe (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32265 cpu_tp ceph NOFD /proc/32235/task/32265/fd (opendir: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32266 cpu_tp ceph cwd unknown /proc/32235/task/32266/cwd (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32266 cpu_tp ceph rtd unknown /proc/32235/task/32266/root (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32266 cpu_tp ceph txt unknown /proc/32235/task/32266/exe (readlink: Permission denied) 2024-03-16T11:08:29.619 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32266 cpu_tp ceph NOFD /proc/32235/task/32266/fd (opendir: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32267 cpu_tp ceph cwd unknown /proc/32235/task/32267/cwd (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32267 cpu_tp ceph rtd unknown /proc/32235/task/32267/root (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32267 cpu_tp ceph txt unknown /proc/32235/task/32267/exe (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32267 cpu_tp ceph NOFD /proc/32235/task/32267/fd (opendir: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32268 cpu_tp ceph cwd unknown /proc/32235/task/32268/cwd (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32268 cpu_tp ceph rtd unknown /proc/32235/task/32268/root (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32268 cpu_tp ceph txt unknown /proc/32235/task/32268/exe (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32268 cpu_tp ceph NOFD /proc/32235/task/32268/fd (opendir: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32269 ms_dispat ceph cwd unknown /proc/32235/task/32269/cwd (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32269 ms_dispat ceph rtd unknown /proc/32235/task/32269/root (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32269 ms_dispat ceph txt unknown /proc/32235/task/32269/exe (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32269 ms_dispat ceph NOFD /proc/32235/task/32269/fd (opendir: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32270 ms_local ceph cwd unknown /proc/32235/task/32270/cwd (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32270 ms_local ceph rtd unknown /proc/32235/task/32270/root (readlink: Permission denied) 2024-03-16T11:08:29.620 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32270 ms_local ceph txt unknown /proc/32235/task/32270/exe (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32270 ms_local ceph NOFD /proc/32235/task/32270/fd (opendir: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32271 safe_time ceph cwd unknown /proc/32235/task/32271/cwd (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32271 safe_time ceph rtd unknown /proc/32235/task/32271/root (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32271 safe_time ceph txt unknown /proc/32235/task/32271/exe (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32271 safe_time ceph NOFD /proc/32235/task/32271/fd (opendir: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32272 ms_dispat ceph cwd unknown /proc/32235/task/32272/cwd (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32272 ms_dispat ceph rtd unknown /proc/32235/task/32272/root (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32272 ms_dispat ceph txt unknown /proc/32235/task/32272/exe (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32272 ms_dispat ceph NOFD /proc/32235/task/32272/fd (opendir: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32273 ms_local ceph cwd unknown /proc/32235/task/32273/cwd (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32273 ms_local ceph rtd unknown /proc/32235/task/32273/root (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32273 ms_local ceph txt unknown /proc/32235/task/32273/exe (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 32273 ms_local ceph NOFD /proc/32235/task/32273/fd (opendir: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 71871 rstore_co ceph cwd unknown /proc/32235/task/71871/cwd (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 71871 rstore_co ceph rtd unknown /proc/32235/task/71871/root (readlink: Permission denied) 2024-03-16T11:08:29.621 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 71871 rstore_co ceph txt unknown /proc/32235/task/71871/exe (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mon 32235 71871 rstore_co ceph NOFD /proc/32235/task/71871/fd (opendir: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32470 root cwd unknown /proc/32470/cwd (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32470 root rtd unknown /proc/32470/root (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32470 root txt unknown /proc/32470/exe (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 32470 root NOFD /proc/32470/fd (opendir: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32472 root cwd unknown /proc/32472/cwd (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32472 root rtd unknown /proc/32472/root (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32472 root txt unknown /proc/32472/exe (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 32472 root NOFD /proc/32472/fd (opendir: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 ceph cwd unknown /proc/32474/cwd (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 ceph rtd unknown /proc/32474/root (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 ceph txt unknown /proc/32474/exe (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 ceph NOFD /proc/32474/fd (opendir: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35008 log ceph cwd unknown /proc/32474/task/35008/cwd (readlink: Permission denied) 2024-03-16T11:08:29.622 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35008 log ceph rtd unknown /proc/32474/task/35008/root (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35008 log ceph txt unknown /proc/32474/task/35008/exe (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35008 log ceph NOFD /proc/32474/task/35008/fd (opendir: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35010 msgr-work ceph cwd unknown /proc/32474/task/35010/cwd (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35010 msgr-work ceph rtd unknown /proc/32474/task/35010/root (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35010 msgr-work ceph txt unknown /proc/32474/task/35010/exe (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35010 msgr-work ceph NOFD /proc/32474/task/35010/fd (opendir: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35011 msgr-work ceph cwd unknown /proc/32474/task/35011/cwd (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35011 msgr-work ceph rtd unknown /proc/32474/task/35011/root (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35011 msgr-work ceph txt unknown /proc/32474/task/35011/exe (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35011 msgr-work ceph NOFD /proc/32474/task/35011/fd (opendir: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35012 msgr-work ceph cwd unknown /proc/32474/task/35012/cwd (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35012 msgr-work ceph rtd unknown /proc/32474/task/35012/root (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35012 msgr-work ceph txt unknown /proc/32474/task/35012/exe (readlink: Permission denied) 2024-03-16T11:08:29.623 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35012 msgr-work ceph NOFD /proc/32474/task/35012/fd (opendir: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35016 service ceph cwd unknown /proc/32474/task/35016/cwd (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35016 service ceph rtd unknown /proc/32474/task/35016/root (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35016 service ceph txt unknown /proc/32474/task/35016/exe (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35016 service ceph NOFD /proc/32474/task/35016/fd (opendir: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35017 admin_soc ceph cwd unknown /proc/32474/task/35017/cwd (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35017 admin_soc ceph rtd unknown /proc/32474/task/35017/root (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35017 admin_soc ceph txt unknown /proc/32474/task/35017/exe (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35017 admin_soc ceph NOFD /proc/32474/task/35017/fd (opendir: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35018 ceph_time ceph cwd unknown /proc/32474/task/35018/cwd (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35018 ceph_time ceph rtd unknown /proc/32474/task/35018/root (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35018 ceph_time ceph txt unknown /proc/32474/task/35018/exe (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35018 ceph_time ceph NOFD /proc/32474/task/35018/fd (opendir: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35019 fn_anonym ceph cwd unknown /proc/32474/task/35019/cwd (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35019 fn_anonym ceph rtd unknown /proc/32474/task/35019/root (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35019 fn_anonym ceph txt unknown /proc/32474/task/35019/exe (readlink: Permission denied) 2024-03-16T11:08:29.624 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35019 fn_anonym ceph NOFD /proc/32474/task/35019/fd (opendir: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35020 signal_ha ceph cwd unknown /proc/32474/task/35020/cwd (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35020 signal_ha ceph rtd unknown /proc/32474/task/35020/root (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35020 signal_ha ceph txt unknown /proc/32474/task/35020/exe (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35020 signal_ha ceph NOFD /proc/32474/task/35020/fd (opendir: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35021 mgrsb-fin ceph cwd unknown /proc/32474/task/35021/cwd (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35021 mgrsb-fin ceph rtd unknown /proc/32474/task/35021/root (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35021 mgrsb-fin ceph txt unknown /proc/32474/task/35021/exe (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35021 mgrsb-fin ceph NOFD /proc/32474/task/35021/fd (opendir: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35022 ms_dispat ceph cwd unknown /proc/32474/task/35022/cwd (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35022 ms_dispat ceph rtd unknown /proc/32474/task/35022/root (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35022 ms_dispat ceph txt unknown /proc/32474/task/35022/exe (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35022 ms_dispat ceph NOFD /proc/32474/task/35022/fd (opendir: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35023 ms_local ceph cwd unknown /proc/32474/task/35023/cwd (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35023 ms_local ceph rtd unknown /proc/32474/task/35023/root (readlink: Permission denied) 2024-03-16T11:08:29.625 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35023 ms_local ceph txt unknown /proc/32474/task/35023/exe (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35023 ms_local ceph NOFD /proc/32474/task/35023/fd (opendir: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35024 io_contex ceph cwd unknown /proc/32474/task/35024/cwd (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35024 io_contex ceph rtd unknown /proc/32474/task/35024/root (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35024 io_contex ceph txt unknown /proc/32474/task/35024/exe (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35024 io_contex ceph NOFD /proc/32474/task/35024/fd (opendir: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35025 io_contex ceph cwd unknown /proc/32474/task/35025/cwd (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35025 io_contex ceph rtd unknown /proc/32474/task/35025/root (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35025 io_contex ceph txt unknown /proc/32474/task/35025/exe (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35025 io_contex ceph NOFD /proc/32474/task/35025/fd (opendir: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35026 safe_time ceph cwd unknown /proc/32474/task/35026/cwd (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35026 safe_time ceph rtd unknown /proc/32474/task/35026/root (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35026 safe_time ceph txt unknown /proc/32474/task/35026/exe (readlink: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35026 safe_time ceph NOFD /proc/32474/task/35026/fd (opendir: Permission denied) 2024-03-16T11:08:29.626 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35027 safe_time ceph cwd unknown /proc/32474/task/35027/cwd (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35027 safe_time ceph rtd unknown /proc/32474/task/35027/root (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35027 safe_time ceph txt unknown /proc/32474/task/35027/exe (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35027 safe_time ceph NOFD /proc/32474/task/35027/fd (opendir: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35028 safe_time ceph cwd unknown /proc/32474/task/35028/cwd (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35028 safe_time ceph rtd unknown /proc/32474/task/35028/root (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35028 safe_time ceph txt unknown /proc/32474/task/35028/exe (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35028 safe_time ceph NOFD /proc/32474/task/35028/fd (opendir: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35029 fn_anonym ceph cwd unknown /proc/32474/task/35029/cwd (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35029 fn_anonym ceph rtd unknown /proc/32474/task/35029/root (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35029 fn_anonym ceph txt unknown /proc/32474/task/35029/exe (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35029 fn_anonym ceph NOFD /proc/32474/task/35029/fd (opendir: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35030 flusher ceph cwd unknown /proc/32474/task/35030/cwd (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35030 flusher ceph rtd unknown /proc/32474/task/35030/root (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35030 flusher ceph txt unknown /proc/32474/task/35030/exe (readlink: Permission denied) 2024-03-16T11:08:29.627 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35030 flusher ceph NOFD /proc/32474/task/35030/fd (opendir: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35031 safe_time ceph cwd unknown /proc/32474/task/35031/cwd (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35031 safe_time ceph rtd unknown /proc/32474/task/35031/root (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35031 safe_time ceph txt unknown /proc/32474/task/35031/exe (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35031 safe_time ceph NOFD /proc/32474/task/35031/fd (opendir: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35200 mgr-fin ceph cwd unknown /proc/32474/task/35200/cwd (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35200 mgr-fin ceph rtd unknown /proc/32474/task/35200/root (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35200 mgr-fin ceph txt unknown /proc/32474/task/35200/exe (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35200 mgr-fin ceph NOFD /proc/32474/task/35200/fd (opendir: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35201 ms_dispat ceph cwd unknown /proc/32474/task/35201/cwd (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35201 ms_dispat ceph rtd unknown /proc/32474/task/35201/root (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35201 ms_dispat ceph txt unknown /proc/32474/task/35201/exe (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35201 ms_dispat ceph NOFD /proc/32474/task/35201/fd (opendir: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35202 ms_local ceph cwd unknown /proc/32474/task/35202/cwd (readlink: Permission denied) 2024-03-16T11:08:29.628 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35202 ms_local ceph rtd unknown /proc/32474/task/35202/root (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35202 ms_local ceph txt unknown /proc/32474/task/35202/exe (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35202 ms_local ceph NOFD /proc/32474/task/35202/fd (opendir: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35203 safe_time ceph cwd unknown /proc/32474/task/35203/cwd (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35203 safe_time ceph rtd unknown /proc/32474/task/35203/root (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35203 safe_time ceph txt unknown /proc/32474/task/35203/exe (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35203 safe_time ceph NOFD /proc/32474/task/35203/fd (opendir: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35204 cmdfin ceph cwd unknown /proc/32474/task/35204/cwd (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35204 cmdfin ceph rtd unknown /proc/32474/task/35204/root (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35204 cmdfin ceph txt unknown /proc/32474/task/35204/exe (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35204 cmdfin ceph NOFD /proc/32474/task/35204/fd (opendir: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35208 io_contex ceph cwd unknown /proc/32474/task/35208/cwd (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35208 io_contex ceph rtd unknown /proc/32474/task/35208/root (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35208 io_contex ceph txt unknown /proc/32474/task/35208/exe (readlink: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35208 io_contex ceph NOFD /proc/32474/task/35208/fd (opendir: Permission denied) 2024-03-16T11:08:29.629 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35209 io_contex ceph cwd unknown /proc/32474/task/35209/cwd (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35209 io_contex ceph rtd unknown /proc/32474/task/35209/root (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35209 io_contex ceph txt unknown /proc/32474/task/35209/exe (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35209 io_contex ceph NOFD /proc/32474/task/35209/fd (opendir: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35210 ceph_time ceph cwd unknown /proc/32474/task/35210/cwd (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35210 ceph_time ceph rtd unknown /proc/32474/task/35210/root (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35210 ceph_time ceph txt unknown /proc/32474/task/35210/exe (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35210 ceph_time ceph NOFD /proc/32474/task/35210/fd (opendir: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35211 ms_dispat ceph cwd unknown /proc/32474/task/35211/cwd (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35211 ms_dispat ceph rtd unknown /proc/32474/task/35211/root (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35211 ms_dispat ceph txt unknown /proc/32474/task/35211/exe (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35211 ms_dispat ceph NOFD /proc/32474/task/35211/fd (opendir: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35212 ms_local ceph cwd unknown /proc/32474/task/35212/cwd (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35212 ms_local ceph rtd unknown /proc/32474/task/35212/root (readlink: Permission denied) 2024-03-16T11:08:29.630 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35212 ms_local ceph txt unknown /proc/32474/task/35212/exe (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35212 ms_local ceph NOFD /proc/32474/task/35212/fd (opendir: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35213 safe_time ceph cwd unknown /proc/32474/task/35213/cwd (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35213 safe_time ceph rtd unknown /proc/32474/task/35213/root (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35213 safe_time ceph txt unknown /proc/32474/task/35213/exe (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35213 safe_time ceph NOFD /proc/32474/task/35213/fd (opendir: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35214 safe_time ceph cwd unknown /proc/32474/task/35214/cwd (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35214 safe_time ceph rtd unknown /proc/32474/task/35214/root (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35214 safe_time ceph txt unknown /proc/32474/task/35214/exe (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35214 safe_time ceph NOFD /proc/32474/task/35214/fd (opendir: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35215 balancer ceph cwd unknown /proc/32474/task/35215/cwd (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35215 balancer ceph rtd unknown /proc/32474/task/35215/root (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35215 balancer ceph txt unknown /proc/32474/task/35215/exe (readlink: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35215 balancer ceph NOFD /proc/32474/task/35215/fd (opendir: Permission denied) 2024-03-16T11:08:29.631 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35216 m-fin-bal ceph cwd unknown /proc/32474/task/35216/cwd (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35216 m-fin-bal ceph rtd unknown /proc/32474/task/35216/root (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35216 m-fin-bal ceph txt unknown /proc/32474/task/35216/exe (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35216 m-fin-bal ceph NOFD /proc/32474/task/35216/fd (opendir: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35217 mgr-fin ceph cwd unknown /proc/32474/task/35217/cwd (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35217 mgr-fin ceph rtd unknown /proc/32474/task/35217/root (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35217 mgr-fin ceph txt unknown /proc/32474/task/35217/exe (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35217 mgr-fin ceph NOFD /proc/32474/task/35217/fd (opendir: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35218 mgr-fin ceph cwd unknown /proc/32474/task/35218/cwd (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35218 mgr-fin ceph rtd unknown /proc/32474/task/35218/root (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35218 mgr-fin ceph txt unknown /proc/32474/task/35218/exe (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35218 mgr-fin ceph NOFD /proc/32474/task/35218/fd (opendir: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35219 mgr-fin ceph cwd unknown /proc/32474/task/35219/cwd (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35219 mgr-fin ceph rtd unknown /proc/32474/task/35219/root (readlink: Permission denied) 2024-03-16T11:08:29.632 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35219 mgr-fin ceph txt unknown /proc/32474/task/35219/exe (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35219 mgr-fin ceph NOFD /proc/32474/task/35219/fd (opendir: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35220 mgr-fin ceph cwd unknown /proc/32474/task/35220/cwd (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35220 mgr-fin ceph rtd unknown /proc/32474/task/35220/root (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35220 mgr-fin ceph txt unknown /proc/32474/task/35220/exe (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35220 mgr-fin ceph NOFD /proc/32474/task/35220/fd (opendir: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35221 mgr-fin ceph cwd unknown /proc/32474/task/35221/cwd (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35221 mgr-fin ceph rtd unknown /proc/32474/task/35221/root (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35221 mgr-fin ceph txt unknown /proc/32474/task/35221/exe (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35221 mgr-fin ceph NOFD /proc/32474/task/35221/fd (opendir: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35222 mgr-fin ceph cwd unknown /proc/32474/task/35222/cwd (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35222 mgr-fin ceph rtd unknown /proc/32474/task/35222/root (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35222 mgr-fin ceph txt unknown /proc/32474/task/35222/exe (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35222 mgr-fin ceph NOFD /proc/32474/task/35222/fd (opendir: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35223 mgr-fin ceph cwd unknown /proc/32474/task/35223/cwd (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35223 mgr-fin ceph rtd unknown /proc/32474/task/35223/root (readlink: Permission denied) 2024-03-16T11:08:29.633 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35223 mgr-fin ceph txt unknown /proc/32474/task/35223/exe (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35223 mgr-fin ceph NOFD /proc/32474/task/35223/fd (opendir: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35224 mgr-fin ceph cwd unknown /proc/32474/task/35224/cwd (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35224 mgr-fin ceph rtd unknown /proc/32474/task/35224/root (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35224 mgr-fin ceph txt unknown /proc/32474/task/35224/exe (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35224 mgr-fin ceph NOFD /proc/32474/task/35224/fd (opendir: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35225 mgr-fin ceph cwd unknown /proc/32474/task/35225/cwd (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35225 mgr-fin ceph rtd unknown /proc/32474/task/35225/root (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35225 mgr-fin ceph txt unknown /proc/32474/task/35225/exe (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35225 mgr-fin ceph NOFD /proc/32474/task/35225/fd (opendir: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35226 mgr-fin ceph cwd unknown /proc/32474/task/35226/cwd (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35226 mgr-fin ceph rtd unknown /proc/32474/task/35226/root (readlink: Permission denied) 2024-03-16T11:08:29.634 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35226 mgr-fin ceph txt unknown /proc/32474/task/35226/exe (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35226 mgr-fin ceph NOFD /proc/32474/task/35226/fd (opendir: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35227 mgr-fin ceph cwd unknown /proc/32474/task/35227/cwd (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35227 mgr-fin ceph rtd unknown /proc/32474/task/35227/root (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35227 mgr-fin ceph txt unknown /proc/32474/task/35227/exe (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35227 mgr-fin ceph NOFD /proc/32474/task/35227/fd (opendir: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35228 mgr-fin ceph cwd unknown /proc/32474/task/35228/cwd (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35228 mgr-fin ceph rtd unknown /proc/32474/task/35228/root (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35228 mgr-fin ceph txt unknown /proc/32474/task/35228/exe (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35228 mgr-fin ceph NOFD /proc/32474/task/35228/fd (opendir: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35229 mgr-fin ceph cwd unknown /proc/32474/task/35229/cwd (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35229 mgr-fin ceph rtd unknown /proc/32474/task/35229/root (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35229 mgr-fin ceph txt unknown /proc/32474/task/35229/exe (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35229 mgr-fin ceph NOFD /proc/32474/task/35229/fd (opendir: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35230 mgr-fin ceph cwd unknown /proc/32474/task/35230/cwd (readlink: Permission denied) 2024-03-16T11:08:29.635 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35230 mgr-fin ceph rtd unknown /proc/32474/task/35230/root (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35230 mgr-fin ceph txt unknown /proc/32474/task/35230/exe (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35230 mgr-fin ceph NOFD /proc/32474/task/35230/fd (opendir: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35231 mgr-fin ceph cwd unknown /proc/32474/task/35231/cwd (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35231 mgr-fin ceph rtd unknown /proc/32474/task/35231/root (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35231 mgr-fin ceph txt unknown /proc/32474/task/35231/exe (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35231 mgr-fin ceph NOFD /proc/32474/task/35231/fd (opendir: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35232 mgr-fin ceph cwd unknown /proc/32474/task/35232/cwd (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35232 mgr-fin ceph rtd unknown /proc/32474/task/35232/root (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35232 mgr-fin ceph txt unknown /proc/32474/task/35232/exe (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35232 mgr-fin ceph NOFD /proc/32474/task/35232/fd (opendir: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35233 cephadm ceph cwd unknown /proc/32474/task/35233/cwd (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35233 cephadm ceph rtd unknown /proc/32474/task/35233/root (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35233 cephadm ceph txt unknown /proc/32474/task/35233/exe (readlink: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35233 cephadm ceph NOFD /proc/32474/task/35233/fd (opendir: Permission denied) 2024-03-16T11:08:29.636 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35234 m-fin-cep ceph cwd unknown /proc/32474/task/35234/cwd (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35234 m-fin-cep ceph rtd unknown /proc/32474/task/35234/root (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35234 m-fin-cep ceph txt unknown /proc/32474/task/35234/exe (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35234 m-fin-cep ceph NOFD /proc/32474/task/35234/fd (opendir: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35235 mgr-fin ceph cwd unknown /proc/32474/task/35235/cwd (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35235 mgr-fin ceph rtd unknown /proc/32474/task/35235/root (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35235 mgr-fin ceph txt unknown /proc/32474/task/35235/exe (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35235 mgr-fin ceph NOFD /proc/32474/task/35235/fd (opendir: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35236 crash ceph cwd unknown /proc/32474/task/35236/cwd (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35236 crash ceph rtd unknown /proc/32474/task/35236/root (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35236 crash ceph txt unknown /proc/32474/task/35236/exe (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35236 crash ceph NOFD /proc/32474/task/35236/fd (opendir: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35237 m-fin-cra ceph cwd unknown /proc/32474/task/35237/cwd (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35237 m-fin-cra ceph rtd unknown /proc/32474/task/35237/root (readlink: Permission denied) 2024-03-16T11:08:29.637 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35237 m-fin-cra ceph txt unknown /proc/32474/task/35237/exe (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35237 m-fin-cra ceph NOFD /proc/32474/task/35237/fd (opendir: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35238 mgr-fin ceph cwd unknown /proc/32474/task/35238/cwd (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35238 mgr-fin ceph rtd unknown /proc/32474/task/35238/root (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35238 mgr-fin ceph txt unknown /proc/32474/task/35238/exe (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35238 mgr-fin ceph NOFD /proc/32474/task/35238/fd (opendir: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35239 cephadm ceph cwd unknown /proc/32474/task/35239/cwd (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35239 cephadm ceph rtd unknown /proc/32474/task/35239/root (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35239 cephadm ceph txt unknown /proc/32474/task/35239/exe (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35239 cephadm ceph NOFD /proc/32474/task/35239/fd (opendir: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35240 mgr-fin ceph cwd unknown /proc/32474/task/35240/cwd (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35240 mgr-fin ceph rtd unknown /proc/32474/task/35240/root (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35240 mgr-fin ceph txt unknown /proc/32474/task/35240/exe (readlink: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35240 mgr-fin ceph NOFD /proc/32474/task/35240/fd (opendir: Permission denied) 2024-03-16T11:08:29.638 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35241 dashboard ceph cwd unknown /proc/32474/task/35241/cwd (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35241 dashboard ceph rtd unknown /proc/32474/task/35241/root (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35241 dashboard ceph txt unknown /proc/32474/task/35241/exe (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35241 dashboard ceph NOFD /proc/32474/task/35241/fd (opendir: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35242 m-fin-das ceph cwd unknown /proc/32474/task/35242/cwd (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35242 m-fin-das ceph rtd unknown /proc/32474/task/35242/root (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35242 m-fin-das ceph txt unknown /proc/32474/task/35242/exe (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35242 m-fin-das ceph NOFD /proc/32474/task/35242/fd (opendir: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35243 mgr-fin ceph cwd unknown /proc/32474/task/35243/cwd (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35243 mgr-fin ceph rtd unknown /proc/32474/task/35243/root (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35243 mgr-fin ceph txt unknown /proc/32474/task/35243/exe (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35243 mgr-fin ceph NOFD /proc/32474/task/35243/fd (opendir: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35244 mgr-fin ceph cwd unknown /proc/32474/task/35244/cwd (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35244 mgr-fin ceph rtd unknown /proc/32474/task/35244/root (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35244 mgr-fin ceph txt unknown /proc/32474/task/35244/exe (readlink: Permission denied) 2024-03-16T11:08:29.639 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35244 mgr-fin ceph NOFD /proc/32474/task/35244/fd (opendir: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35245 devicehea ceph cwd unknown /proc/32474/task/35245/cwd (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35245 devicehea ceph rtd unknown /proc/32474/task/35245/root (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35245 devicehea ceph txt unknown /proc/32474/task/35245/exe (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35245 devicehea ceph NOFD /proc/32474/task/35245/fd (opendir: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35246 m-fin-dev ceph cwd unknown /proc/32474/task/35246/cwd (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35246 m-fin-dev ceph rtd unknown /proc/32474/task/35246/root (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35246 m-fin-dev ceph txt unknown /proc/32474/task/35246/exe (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35246 m-fin-dev ceph NOFD /proc/32474/task/35246/fd (opendir: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35247 mgr-fin ceph cwd unknown /proc/32474/task/35247/cwd (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35247 mgr-fin ceph rtd unknown /proc/32474/task/35247/root (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35247 mgr-fin ceph txt unknown /proc/32474/task/35247/exe (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35247 mgr-fin ceph NOFD /proc/32474/task/35247/fd (opendir: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35248 mgr-fin ceph cwd unknown /proc/32474/task/35248/cwd (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35248 mgr-fin ceph rtd unknown /proc/32474/task/35248/root (readlink: Permission denied) 2024-03-16T11:08:29.640 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35248 mgr-fin ceph txt unknown /proc/32474/task/35248/exe (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35248 mgr-fin ceph NOFD /proc/32474/task/35248/fd (opendir: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35249 mgr-fin ceph cwd unknown /proc/32474/task/35249/cwd (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35249 mgr-fin ceph rtd unknown /proc/32474/task/35249/root (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35249 mgr-fin ceph txt unknown /proc/32474/task/35249/exe (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35249 mgr-fin ceph NOFD /proc/32474/task/35249/fd (opendir: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35251 m-fin-ios ceph cwd unknown /proc/32474/task/35251/cwd (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35251 m-fin-ios ceph rtd unknown /proc/32474/task/35251/root (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35251 m-fin-ios ceph txt unknown /proc/32474/task/35251/exe (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35251 m-fin-ios ceph NOFD /proc/32474/task/35251/fd (opendir: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35252 mgr-fin ceph cwd unknown /proc/32474/task/35252/cwd (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35252 mgr-fin ceph rtd unknown /proc/32474/task/35252/root (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35252 mgr-fin ceph txt unknown /proc/32474/task/35252/exe (readlink: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35252 mgr-fin ceph NOFD /proc/32474/task/35252/fd (opendir: Permission denied) 2024-03-16T11:08:29.641 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35254 m-fin-nfs ceph cwd unknown /proc/32474/task/35254/cwd (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35254 m-fin-nfs ceph rtd unknown /proc/32474/task/35254/root (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35254 m-fin-nfs ceph txt unknown /proc/32474/task/35254/exe (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35254 m-fin-nfs ceph NOFD /proc/32474/task/35254/fd (opendir: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35255 mgr-fin ceph cwd unknown /proc/32474/task/35255/cwd (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35255 mgr-fin ceph rtd unknown /proc/32474/task/35255/root (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35255 mgr-fin ceph txt unknown /proc/32474/task/35255/exe (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35255 mgr-fin ceph NOFD /proc/32474/task/35255/fd (opendir: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35256 mgr-fin ceph cwd unknown /proc/32474/task/35256/cwd (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35256 mgr-fin ceph rtd unknown /proc/32474/task/35256/root (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35256 mgr-fin ceph txt unknown /proc/32474/task/35256/exe (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35256 mgr-fin ceph NOFD /proc/32474/task/35256/fd (opendir: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35257 mgr-fin ceph cwd unknown /proc/32474/task/35257/cwd (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35257 mgr-fin ceph rtd unknown /proc/32474/task/35257/root (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35257 mgr-fin ceph txt unknown /proc/32474/task/35257/exe (readlink: Permission denied) 2024-03-16T11:08:29.642 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35257 mgr-fin ceph NOFD /proc/32474/task/35257/fd (opendir: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35259 m-fin-orc ceph cwd unknown /proc/32474/task/35259/cwd (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35259 m-fin-orc ceph rtd unknown /proc/32474/task/35259/root (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35259 m-fin-orc ceph txt unknown /proc/32474/task/35259/exe (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35259 m-fin-orc ceph NOFD /proc/32474/task/35259/fd (opendir: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35260 pg_autosc ceph cwd unknown /proc/32474/task/35260/cwd (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35260 pg_autosc ceph rtd unknown /proc/32474/task/35260/root (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35260 pg_autosc ceph txt unknown /proc/32474/task/35260/exe (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35260 pg_autosc ceph NOFD /proc/32474/task/35260/fd (opendir: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35261 m-fin-pg_ ceph cwd unknown /proc/32474/task/35261/cwd (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35261 m-fin-pg_ ceph rtd unknown /proc/32474/task/35261/root (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35261 m-fin-pg_ ceph txt unknown /proc/32474/task/35261/exe (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35261 m-fin-pg_ ceph NOFD /proc/32474/task/35261/fd (opendir: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35262 progress ceph cwd unknown /proc/32474/task/35262/cwd (readlink: Permission denied) 2024-03-16T11:08:29.643 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35262 progress ceph rtd unknown /proc/32474/task/35262/root (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35262 progress ceph txt unknown /proc/32474/task/35262/exe (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35262 progress ceph NOFD /proc/32474/task/35262/fd (opendir: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35263 m-fin-pro ceph cwd unknown /proc/32474/task/35263/cwd (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35263 m-fin-pro ceph rtd unknown /proc/32474/task/35263/root (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35263 m-fin-pro ceph txt unknown /proc/32474/task/35263/exe (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35263 m-fin-pro ceph NOFD /proc/32474/task/35263/fd (opendir: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35267 io_contex ceph cwd unknown /proc/32474/task/35267/cwd (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35267 io_contex ceph rtd unknown /proc/32474/task/35267/root (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35267 io_contex ceph txt unknown /proc/32474/task/35267/exe (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35267 io_contex ceph NOFD /proc/32474/task/35267/fd (opendir: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35268 io_contex ceph cwd unknown /proc/32474/task/35268/cwd (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35268 io_contex ceph rtd unknown /proc/32474/task/35268/root (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35268 io_contex ceph txt unknown /proc/32474/task/35268/exe (readlink: Permission denied) 2024-03-16T11:08:29.644 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35268 io_contex ceph NOFD /proc/32474/task/35268/fd (opendir: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35269 ceph_time ceph cwd unknown /proc/32474/task/35269/cwd (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35269 ceph_time ceph rtd unknown /proc/32474/task/35269/root (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35269 ceph_time ceph txt unknown /proc/32474/task/35269/exe (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35269 ceph_time ceph NOFD /proc/32474/task/35269/fd (opendir: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35270 ms_dispat ceph cwd unknown /proc/32474/task/35270/cwd (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35270 ms_dispat ceph rtd unknown /proc/32474/task/35270/root (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35270 ms_dispat ceph txt unknown /proc/32474/task/35270/exe (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35270 ms_dispat ceph NOFD /proc/32474/task/35270/fd (opendir: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35271 ms_local ceph cwd unknown /proc/32474/task/35271/cwd (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35271 ms_local ceph rtd unknown /proc/32474/task/35271/root (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35271 ms_local ceph txt unknown /proc/32474/task/35271/exe (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35271 ms_local ceph NOFD /proc/32474/task/35271/fd (opendir: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35272 safe_time ceph cwd unknown /proc/32474/task/35272/cwd (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35272 safe_time ceph rtd unknown /proc/32474/task/35272/root (readlink: Permission denied) 2024-03-16T11:08:29.645 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35272 safe_time ceph txt unknown /proc/32474/task/35272/exe (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35272 safe_time ceph NOFD /proc/32474/task/35272/fd (opendir: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35273 safe_time ceph cwd unknown /proc/32474/task/35273/cwd (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35273 safe_time ceph rtd unknown /proc/32474/task/35273/root (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35273 safe_time ceph txt unknown /proc/32474/task/35273/exe (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35273 safe_time ceph NOFD /proc/32474/task/35273/fd (opendir: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35274 mgr-fin ceph cwd unknown /proc/32474/task/35274/cwd (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35274 mgr-fin ceph rtd unknown /proc/32474/task/35274/root (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35274 mgr-fin ceph txt unknown /proc/32474/task/35274/exe (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35274 mgr-fin ceph NOFD /proc/32474/task/35274/fd (opendir: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35276 m-fin-rbd ceph cwd unknown /proc/32474/task/35276/cwd (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35276 m-fin-rbd ceph rtd unknown /proc/32474/task/35276/root (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35276 m-fin-rbd ceph txt unknown /proc/32474/task/35276/exe (readlink: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35276 m-fin-rbd ceph NOFD /proc/32474/task/35276/fd (opendir: Permission denied) 2024-03-16T11:08:29.646 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35277 restful ceph cwd unknown /proc/32474/task/35277/cwd (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35277 restful ceph rtd unknown /proc/32474/task/35277/root (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35277 restful ceph txt unknown /proc/32474/task/35277/exe (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35277 restful ceph NOFD /proc/32474/task/35277/fd (opendir: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35278 m-fin-res ceph cwd unknown /proc/32474/task/35278/cwd (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35278 m-fin-res ceph rtd unknown /proc/32474/task/35278/root (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35278 m-fin-res ceph txt unknown /proc/32474/task/35278/exe (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35278 m-fin-res ceph NOFD /proc/32474/task/35278/fd (opendir: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35280 m-fin-sta ceph cwd unknown /proc/32474/task/35280/cwd (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35280 m-fin-sta ceph rtd unknown /proc/32474/task/35280/root (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35280 m-fin-sta ceph txt unknown /proc/32474/task/35280/exe (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35280 m-fin-sta ceph NOFD /proc/32474/task/35280/fd (opendir: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35281 mgr-fin ceph cwd unknown /proc/32474/task/35281/cwd (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35281 mgr-fin ceph rtd unknown /proc/32474/task/35281/root (readlink: Permission denied) 2024-03-16T11:08:29.647 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35281 mgr-fin ceph txt unknown /proc/32474/task/35281/exe (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35281 mgr-fin ceph NOFD /proc/32474/task/35281/fd (opendir: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35282 mgr-fin ceph cwd unknown /proc/32474/task/35282/cwd (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35282 mgr-fin ceph rtd unknown /proc/32474/task/35282/root (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35282 mgr-fin ceph txt unknown /proc/32474/task/35282/exe (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35282 mgr-fin ceph NOFD /proc/32474/task/35282/fd (opendir: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35283 mgr-fin ceph cwd unknown /proc/32474/task/35283/cwd (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35283 mgr-fin ceph rtd unknown /proc/32474/task/35283/root (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35283 mgr-fin ceph txt unknown /proc/32474/task/35283/exe (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35283 mgr-fin ceph NOFD /proc/32474/task/35283/fd (opendir: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35284 telemetry ceph cwd unknown /proc/32474/task/35284/cwd (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35284 telemetry ceph rtd unknown /proc/32474/task/35284/root (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35284 telemetry ceph txt unknown /proc/32474/task/35284/exe (readlink: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35284 telemetry ceph NOFD /proc/32474/task/35284/fd (opendir: Permission denied) 2024-03-16T11:08:29.648 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35285 m-fin-tel ceph cwd unknown /proc/32474/task/35285/cwd (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35285 m-fin-tel ceph rtd unknown /proc/32474/task/35285/root (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35285 m-fin-tel ceph txt unknown /proc/32474/task/35285/exe (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35285 m-fin-tel ceph NOFD /proc/32474/task/35285/fd (opendir: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35286 mgr-fin ceph cwd unknown /proc/32474/task/35286/cwd (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35286 mgr-fin ceph rtd unknown /proc/32474/task/35286/root (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35286 mgr-fin ceph txt unknown /proc/32474/task/35286/exe (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35286 mgr-fin ceph NOFD /proc/32474/task/35286/fd (opendir: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35287 mgr-fin ceph cwd unknown /proc/32474/task/35287/cwd (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35287 mgr-fin ceph rtd unknown /proc/32474/task/35287/root (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35287 mgr-fin ceph txt unknown /proc/32474/task/35287/exe (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35287 mgr-fin ceph NOFD /proc/32474/task/35287/fd (opendir: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35292 io_contex ceph cwd unknown /proc/32474/task/35292/cwd (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35292 io_contex ceph rtd unknown /proc/32474/task/35292/root (readlink: Permission denied) 2024-03-16T11:08:29.649 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35292 io_contex ceph txt unknown /proc/32474/task/35292/exe (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35292 io_contex ceph NOFD /proc/32474/task/35292/fd (opendir: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35293 io_contex ceph cwd unknown /proc/32474/task/35293/cwd (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35293 io_contex ceph rtd unknown /proc/32474/task/35293/root (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35293 io_contex ceph txt unknown /proc/32474/task/35293/exe (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35293 io_contex ceph NOFD /proc/32474/task/35293/fd (opendir: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35294 ceph_time ceph cwd unknown /proc/32474/task/35294/cwd (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35294 ceph_time ceph rtd unknown /proc/32474/task/35294/root (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35294 ceph_time ceph txt unknown /proc/32474/task/35294/exe (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35294 ceph_time ceph NOFD /proc/32474/task/35294/fd (opendir: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35295 ms_dispat ceph cwd unknown /proc/32474/task/35295/cwd (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35295 ms_dispat ceph rtd unknown /proc/32474/task/35295/root (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35295 ms_dispat ceph txt unknown /proc/32474/task/35295/exe (readlink: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35295 ms_dispat ceph NOFD /proc/32474/task/35295/fd (opendir: Permission denied) 2024-03-16T11:08:29.650 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35296 ms_local ceph cwd unknown /proc/32474/task/35296/cwd (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35296 ms_local ceph rtd unknown /proc/32474/task/35296/root (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35296 ms_local ceph txt unknown /proc/32474/task/35296/exe (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35296 ms_local ceph NOFD /proc/32474/task/35296/fd (opendir: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35297 safe_time ceph cwd unknown /proc/32474/task/35297/cwd (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35297 safe_time ceph rtd unknown /proc/32474/task/35297/root (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35297 safe_time ceph txt unknown /proc/32474/task/35297/exe (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35297 safe_time ceph NOFD /proc/32474/task/35297/fd (opendir: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35298 safe_time ceph cwd unknown /proc/32474/task/35298/cwd (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35298 safe_time ceph rtd unknown /proc/32474/task/35298/root (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35298 safe_time ceph txt unknown /proc/32474/task/35298/exe (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35298 safe_time ceph NOFD /proc/32474/task/35298/fd (opendir: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35299 mgr-fin ceph cwd unknown /proc/32474/task/35299/cwd (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35299 mgr-fin ceph rtd unknown /proc/32474/task/35299/root (readlink: Permission denied) 2024-03-16T11:08:29.651 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35299 mgr-fin ceph txt unknown /proc/32474/task/35299/exe (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35299 mgr-fin ceph NOFD /proc/32474/task/35299/fd (opendir: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35300 mgr-fin ceph cwd unknown /proc/32474/task/35300/cwd (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35300 mgr-fin ceph rtd unknown /proc/32474/task/35300/root (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35300 mgr-fin ceph txt unknown /proc/32474/task/35300/exe (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35300 mgr-fin ceph NOFD /proc/32474/task/35300/fd (opendir: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35301 mgr-fin ceph cwd unknown /proc/32474/task/35301/cwd (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35301 mgr-fin ceph rtd unknown /proc/32474/task/35301/root (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35301 mgr-fin ceph txt unknown /proc/32474/task/35301/exe (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35301 mgr-fin ceph NOFD /proc/32474/task/35301/fd (opendir: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35302 mgr-fin ceph cwd unknown /proc/32474/task/35302/cwd (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35302 mgr-fin ceph rtd unknown /proc/32474/task/35302/root (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35302 mgr-fin ceph txt unknown /proc/32474/task/35302/exe (readlink: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35302 mgr-fin ceph NOFD /proc/32474/task/35302/fd (opendir: Permission denied) 2024-03-16T11:08:29.652 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35303 mgr-fin ceph cwd unknown /proc/32474/task/35303/cwd (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35303 mgr-fin ceph rtd unknown /proc/32474/task/35303/root (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35303 mgr-fin ceph txt unknown /proc/32474/task/35303/exe (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35303 mgr-fin ceph NOFD /proc/32474/task/35303/fd (opendir: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35304 mgr-fin ceph cwd unknown /proc/32474/task/35304/cwd (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35304 mgr-fin ceph rtd unknown /proc/32474/task/35304/root (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35304 mgr-fin ceph txt unknown /proc/32474/task/35304/exe (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35304 mgr-fin ceph NOFD /proc/32474/task/35304/fd (opendir: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35305 mgr-fin ceph cwd unknown /proc/32474/task/35305/cwd (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35305 mgr-fin ceph rtd unknown /proc/32474/task/35305/root (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35305 mgr-fin ceph txt unknown /proc/32474/task/35305/exe (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35305 mgr-fin ceph NOFD /proc/32474/task/35305/fd (opendir: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35306 mgr-fin ceph cwd unknown /proc/32474/task/35306/cwd (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35306 mgr-fin ceph rtd unknown /proc/32474/task/35306/root (readlink: Permission denied) 2024-03-16T11:08:29.653 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35306 mgr-fin ceph txt unknown /proc/32474/task/35306/exe (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35306 mgr-fin ceph NOFD /proc/32474/task/35306/fd (opendir: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35307 mgr-fin ceph cwd unknown /proc/32474/task/35307/cwd (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35307 mgr-fin ceph rtd unknown /proc/32474/task/35307/root (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35307 mgr-fin ceph txt unknown /proc/32474/task/35307/exe (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35307 mgr-fin ceph NOFD /proc/32474/task/35307/fd (opendir: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35308 mgr-fin ceph cwd unknown /proc/32474/task/35308/cwd (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35308 mgr-fin ceph rtd unknown /proc/32474/task/35308/root (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35308 mgr-fin ceph txt unknown /proc/32474/task/35308/exe (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35308 mgr-fin ceph NOFD /proc/32474/task/35308/fd (opendir: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35309 mgr-fin ceph cwd unknown /proc/32474/task/35309/cwd (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35309 mgr-fin ceph rtd unknown /proc/32474/task/35309/root (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35309 mgr-fin ceph txt unknown /proc/32474/task/35309/exe (readlink: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35309 mgr-fin ceph NOFD /proc/32474/task/35309/fd (opendir: Permission denied) 2024-03-16T11:08:29.654 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35310 mgr-fin ceph cwd unknown /proc/32474/task/35310/cwd (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35310 mgr-fin ceph rtd unknown /proc/32474/task/35310/root (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35310 mgr-fin ceph txt unknown /proc/32474/task/35310/exe (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35310 mgr-fin ceph NOFD /proc/32474/task/35310/fd (opendir: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35312 m-fin-vol ceph cwd unknown /proc/32474/task/35312/cwd (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35312 m-fin-vol ceph rtd unknown /proc/32474/task/35312/root (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35312 m-fin-vol ceph txt unknown /proc/32474/task/35312/exe (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35312 m-fin-vol ceph NOFD /proc/32474/task/35312/fd (opendir: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35382 mgr-fin ceph cwd unknown /proc/32474/task/35382/cwd (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35382 mgr-fin ceph rtd unknown /proc/32474/task/35382/root (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35382 mgr-fin ceph txt unknown /proc/32474/task/35382/exe (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35382 mgr-fin ceph NOFD /proc/32474/task/35382/fd (opendir: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35383 mgr-fin ceph cwd unknown /proc/32474/task/35383/cwd (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35383 mgr-fin ceph rtd unknown /proc/32474/task/35383/root (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35383 mgr-fin ceph txt unknown /proc/32474/task/35383/exe (readlink: Permission denied) 2024-03-16T11:08:29.655 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35383 mgr-fin ceph NOFD /proc/32474/task/35383/fd (opendir: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35384 mgr-fin ceph cwd unknown /proc/32474/task/35384/cwd (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35384 mgr-fin ceph rtd unknown /proc/32474/task/35384/root (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35384 mgr-fin ceph txt unknown /proc/32474/task/35384/exe (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35384 mgr-fin ceph NOFD /proc/32474/task/35384/fd (opendir: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35385 mgr-fin ceph cwd unknown /proc/32474/task/35385/cwd (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35385 mgr-fin ceph rtd unknown /proc/32474/task/35385/root (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35385 mgr-fin ceph txt unknown /proc/32474/task/35385/exe (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35385 mgr-fin ceph NOFD /proc/32474/task/35385/fd (opendir: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35386 mgr-fin ceph cwd unknown /proc/32474/task/35386/cwd (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35386 mgr-fin ceph rtd unknown /proc/32474/task/35386/root (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35386 mgr-fin ceph txt unknown /proc/32474/task/35386/exe (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35386 mgr-fin ceph NOFD /proc/32474/task/35386/fd (opendir: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35387 mgr-fin ceph cwd unknown /proc/32474/task/35387/cwd (readlink: Permission denied) 2024-03-16T11:08:29.656 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35387 mgr-fin ceph rtd unknown /proc/32474/task/35387/root (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35387 mgr-fin ceph txt unknown /proc/32474/task/35387/exe (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35387 mgr-fin ceph NOFD /proc/32474/task/35387/fd (opendir: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35388 mgr-fin ceph cwd unknown /proc/32474/task/35388/cwd (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35388 mgr-fin ceph rtd unknown /proc/32474/task/35388/root (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35388 mgr-fin ceph txt unknown /proc/32474/task/35388/exe (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35388 mgr-fin ceph NOFD /proc/32474/task/35388/fd (opendir: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35389 mgr-fin ceph cwd unknown /proc/32474/task/35389/cwd (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35389 mgr-fin ceph rtd unknown /proc/32474/task/35389/root (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35389 mgr-fin ceph txt unknown /proc/32474/task/35389/exe (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35389 mgr-fin ceph NOFD /proc/32474/task/35389/fd (opendir: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35390 mgr-fin ceph cwd unknown /proc/32474/task/35390/cwd (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35390 mgr-fin ceph rtd unknown /proc/32474/task/35390/root (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35390 mgr-fin ceph txt unknown /proc/32474/task/35390/exe (readlink: Permission denied) 2024-03-16T11:08:29.657 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35390 mgr-fin ceph NOFD /proc/32474/task/35390/fd (opendir: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35391 mgr-fin ceph cwd unknown /proc/32474/task/35391/cwd (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35391 mgr-fin ceph rtd unknown /proc/32474/task/35391/root (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35391 mgr-fin ceph txt unknown /proc/32474/task/35391/exe (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35391 mgr-fin ceph NOFD /proc/32474/task/35391/fd (opendir: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35392 mgr-fin ceph cwd unknown /proc/32474/task/35392/cwd (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35392 mgr-fin ceph rtd unknown /proc/32474/task/35392/root (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35392 mgr-fin ceph txt unknown /proc/32474/task/35392/exe (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35392 mgr-fin ceph NOFD /proc/32474/task/35392/fd (opendir: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35420 mgr-fin ceph cwd unknown /proc/32474/task/35420/cwd (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35420 mgr-fin ceph rtd unknown /proc/32474/task/35420/root (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35420 mgr-fin ceph txt unknown /proc/32474/task/35420/exe (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35420 mgr-fin ceph NOFD /proc/32474/task/35420/fd (opendir: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35421 mgr-fin ceph cwd unknown /proc/32474/task/35421/cwd (readlink: Permission denied) 2024-03-16T11:08:29.658 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35421 mgr-fin ceph rtd unknown /proc/32474/task/35421/root (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35421 mgr-fin ceph txt unknown /proc/32474/task/35421/exe (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35421 mgr-fin ceph NOFD /proc/32474/task/35421/fd (opendir: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35422 mgr-fin ceph cwd unknown /proc/32474/task/35422/cwd (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35422 mgr-fin ceph rtd unknown /proc/32474/task/35422/root (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35422 mgr-fin ceph txt unknown /proc/32474/task/35422/exe (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35422 mgr-fin ceph NOFD /proc/32474/task/35422/fd (opendir: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35423 mgr-fin ceph cwd unknown /proc/32474/task/35423/cwd (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35423 mgr-fin ceph rtd unknown /proc/32474/task/35423/root (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35423 mgr-fin ceph txt unknown /proc/32474/task/35423/exe (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35423 mgr-fin ceph NOFD /proc/32474/task/35423/fd (opendir: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35424 mgr-fin ceph cwd unknown /proc/32474/task/35424/cwd (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35424 mgr-fin ceph rtd unknown /proc/32474/task/35424/root (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35424 mgr-fin ceph txt unknown /proc/32474/task/35424/exe (readlink: Permission denied) 2024-03-16T11:08:29.659 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35424 mgr-fin ceph NOFD /proc/32474/task/35424/fd (opendir: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35425 mgr-fin ceph cwd unknown /proc/32474/task/35425/cwd (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35425 mgr-fin ceph rtd unknown /proc/32474/task/35425/root (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35425 mgr-fin ceph txt unknown /proc/32474/task/35425/exe (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35425 mgr-fin ceph NOFD /proc/32474/task/35425/fd (opendir: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35426 mgr-fin ceph cwd unknown /proc/32474/task/35426/cwd (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35426 mgr-fin ceph rtd unknown /proc/32474/task/35426/root (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35426 mgr-fin ceph txt unknown /proc/32474/task/35426/exe (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35426 mgr-fin ceph NOFD /proc/32474/task/35426/fd (opendir: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35427 mgr-fin ceph cwd unknown /proc/32474/task/35427/cwd (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35427 mgr-fin ceph rtd unknown /proc/32474/task/35427/root (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35427 mgr-fin ceph txt unknown /proc/32474/task/35427/exe (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35427 mgr-fin ceph NOFD /proc/32474/task/35427/fd (opendir: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35428 mgr-fin ceph cwd unknown /proc/32474/task/35428/cwd (readlink: Permission denied) 2024-03-16T11:08:29.660 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35428 mgr-fin ceph rtd unknown /proc/32474/task/35428/root (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35428 mgr-fin ceph txt unknown /proc/32474/task/35428/exe (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35428 mgr-fin ceph NOFD /proc/32474/task/35428/fd (opendir: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35429 mgr-fin ceph cwd unknown /proc/32474/task/35429/cwd (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35429 mgr-fin ceph rtd unknown /proc/32474/task/35429/root (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35429 mgr-fin ceph txt unknown /proc/32474/task/35429/exe (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35429 mgr-fin ceph NOFD /proc/32474/task/35429/fd (opendir: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35430 mgr-fin ceph cwd unknown /proc/32474/task/35430/cwd (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35430 mgr-fin ceph rtd unknown /proc/32474/task/35430/root (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35430 mgr-fin ceph txt unknown /proc/32474/task/35430/exe (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 35430 mgr-fin ceph NOFD /proc/32474/task/35430/fd (opendir: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36009 dashboard ceph cwd unknown /proc/32474/task/36009/cwd (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36009 dashboard ceph rtd unknown /proc/32474/task/36009/root (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36009 dashboard ceph txt unknown /proc/32474/task/36009/exe (readlink: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36009 dashboard ceph NOFD /proc/32474/task/36009/fd (opendir: Permission denied) 2024-03-16T11:08:29.661 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36011 dashboard ceph cwd unknown /proc/32474/task/36011/cwd (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36011 dashboard ceph rtd unknown /proc/32474/task/36011/root (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36011 dashboard ceph txt unknown /proc/32474/task/36011/exe (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36011 dashboard ceph NOFD /proc/32474/task/36011/fd (opendir: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36012 dashboard ceph cwd unknown /proc/32474/task/36012/cwd (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36012 dashboard ceph rtd unknown /proc/32474/task/36012/root (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36012 dashboard ceph txt unknown /proc/32474/task/36012/exe (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36012 dashboard ceph NOFD /proc/32474/task/36012/fd (opendir: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36013 dashboard ceph cwd unknown /proc/32474/task/36013/cwd (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36013 dashboard ceph rtd unknown /proc/32474/task/36013/root (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36013 dashboard ceph txt unknown /proc/32474/task/36013/exe (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36013 dashboard ceph NOFD /proc/32474/task/36013/fd (opendir: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36014 dashboard ceph cwd unknown /proc/32474/task/36014/cwd (readlink: Permission denied) 2024-03-16T11:08:29.662 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36014 dashboard ceph rtd unknown /proc/32474/task/36014/root (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36014 dashboard ceph txt unknown /proc/32474/task/36014/exe (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36014 dashboard ceph NOFD /proc/32474/task/36014/fd (opendir: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36015 dashboard ceph cwd unknown /proc/32474/task/36015/cwd (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36015 dashboard ceph rtd unknown /proc/32474/task/36015/root (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36015 dashboard ceph txt unknown /proc/32474/task/36015/exe (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36015 dashboard ceph NOFD /proc/32474/task/36015/fd (opendir: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36016 dashboard ceph cwd unknown /proc/32474/task/36016/cwd (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36016 dashboard ceph rtd unknown /proc/32474/task/36016/root (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36016 dashboard ceph txt unknown /proc/32474/task/36016/exe (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36016 dashboard ceph NOFD /proc/32474/task/36016/fd (opendir: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36017 dashboard ceph cwd unknown /proc/32474/task/36017/cwd (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36017 dashboard ceph rtd unknown /proc/32474/task/36017/root (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36017 dashboard ceph txt unknown /proc/32474/task/36017/exe (readlink: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36017 dashboard ceph NOFD /proc/32474/task/36017/fd (opendir: Permission denied) 2024-03-16T11:08:29.663 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36018 dashboard ceph cwd unknown /proc/32474/task/36018/cwd (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36018 dashboard ceph rtd unknown /proc/32474/task/36018/root (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36018 dashboard ceph txt unknown /proc/32474/task/36018/exe (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36018 dashboard ceph NOFD /proc/32474/task/36018/fd (opendir: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36019 dashboard ceph cwd unknown /proc/32474/task/36019/cwd (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36019 dashboard ceph rtd unknown /proc/32474/task/36019/root (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36019 dashboard ceph txt unknown /proc/32474/task/36019/exe (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36019 dashboard ceph NOFD /proc/32474/task/36019/fd (opendir: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36020 dashboard ceph cwd unknown /proc/32474/task/36020/cwd (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36020 dashboard ceph rtd unknown /proc/32474/task/36020/root (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36020 dashboard ceph txt unknown /proc/32474/task/36020/exe (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36020 dashboard ceph NOFD /proc/32474/task/36020/fd (opendir: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36027 dashboard ceph cwd unknown /proc/32474/task/36027/cwd (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36027 dashboard ceph rtd unknown /proc/32474/task/36027/root (readlink: Permission denied) 2024-03-16T11:08:29.664 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36027 dashboard ceph txt unknown /proc/32474/task/36027/exe (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 36027 dashboard ceph NOFD /proc/32474/task/36027/fd (opendir: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42270 cephadm ceph cwd unknown /proc/32474/task/42270/cwd (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42270 cephadm ceph rtd unknown /proc/32474/task/42270/root (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42270 cephadm ceph txt unknown /proc/32474/task/42270/exe (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42270 cephadm ceph NOFD /proc/32474/task/42270/fd (opendir: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42271 cephadm ceph cwd unknown /proc/32474/task/42271/cwd (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42271 cephadm ceph rtd unknown /proc/32474/task/42271/root (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42271 cephadm ceph txt unknown /proc/32474/task/42271/exe (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42271 cephadm ceph NOFD /proc/32474/task/42271/fd (opendir: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42272 cephadm ceph cwd unknown /proc/32474/task/42272/cwd (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42272 cephadm ceph rtd unknown /proc/32474/task/42272/root (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42272 cephadm ceph txt unknown /proc/32474/task/42272/exe (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42272 cephadm ceph NOFD /proc/32474/task/42272/fd (opendir: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42273 cephadm ceph cwd unknown /proc/32474/task/42273/cwd (readlink: Permission denied) 2024-03-16T11:08:29.665 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42273 cephadm ceph rtd unknown /proc/32474/task/42273/root (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42273 cephadm ceph txt unknown /proc/32474/task/42273/exe (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42273 cephadm ceph NOFD /proc/32474/task/42273/fd (opendir: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42274 cephadm ceph cwd unknown /proc/32474/task/42274/cwd (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42274 cephadm ceph rtd unknown /proc/32474/task/42274/root (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42274 cephadm ceph txt unknown /proc/32474/task/42274/exe (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42274 cephadm ceph NOFD /proc/32474/task/42274/fd (opendir: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42275 cephadm ceph cwd unknown /proc/32474/task/42275/cwd (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42275 cephadm ceph rtd unknown /proc/32474/task/42275/root (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42275 cephadm ceph txt unknown /proc/32474/task/42275/exe (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42275 cephadm ceph NOFD /proc/32474/task/42275/fd (opendir: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42276 cephadm ceph cwd unknown /proc/32474/task/42276/cwd (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42276 cephadm ceph rtd unknown /proc/32474/task/42276/root (readlink: Permission denied) 2024-03-16T11:08:29.666 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42276 cephadm ceph txt unknown /proc/32474/task/42276/exe (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42276 cephadm ceph NOFD /proc/32474/task/42276/fd (opendir: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42277 cephadm ceph cwd unknown /proc/32474/task/42277/cwd (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42277 cephadm ceph rtd unknown /proc/32474/task/42277/root (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42277 cephadm ceph txt unknown /proc/32474/task/42277/exe (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42277 cephadm ceph NOFD /proc/32474/task/42277/fd (opendir: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42278 cephadm ceph cwd unknown /proc/32474/task/42278/cwd (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42278 cephadm ceph rtd unknown /proc/32474/task/42278/root (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42278 cephadm ceph txt unknown /proc/32474/task/42278/exe (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42278 cephadm ceph NOFD /proc/32474/task/42278/fd (opendir: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42279 cephadm ceph cwd unknown /proc/32474/task/42279/cwd (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42279 cephadm ceph rtd unknown /proc/32474/task/42279/root (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42279 cephadm ceph txt unknown /proc/32474/task/42279/exe (readlink: Permission denied) 2024-03-16T11:08:29.667 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42279 cephadm ceph NOFD /proc/32474/task/42279/fd (opendir: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42595 cephadm ceph cwd unknown /proc/32474/task/42595/cwd (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42595 cephadm ceph rtd unknown /proc/32474/task/42595/root (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42595 cephadm ceph txt unknown /proc/32474/task/42595/exe (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42595 cephadm ceph NOFD /proc/32474/task/42595/fd (opendir: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42596 cephadm ceph cwd unknown /proc/32474/task/42596/cwd (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42596 cephadm ceph rtd unknown /proc/32474/task/42596/root (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42596 cephadm ceph txt unknown /proc/32474/task/42596/exe (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42596 cephadm ceph NOFD /proc/32474/task/42596/fd (opendir: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42597 cephadm ceph cwd unknown /proc/32474/task/42597/cwd (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42597 cephadm ceph rtd unknown /proc/32474/task/42597/root (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42597 cephadm ceph txt unknown /proc/32474/task/42597/exe (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42597 cephadm ceph NOFD /proc/32474/task/42597/fd (opendir: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42598 cephadm ceph cwd unknown /proc/32474/task/42598/cwd (readlink: Permission denied) 2024-03-16T11:08:29.668 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42598 cephadm ceph rtd unknown /proc/32474/task/42598/root (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42598 cephadm ceph txt unknown /proc/32474/task/42598/exe (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42598 cephadm ceph NOFD /proc/32474/task/42598/fd (opendir: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42599 cephadm ceph cwd unknown /proc/32474/task/42599/cwd (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42599 cephadm ceph rtd unknown /proc/32474/task/42599/root (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42599 cephadm ceph txt unknown /proc/32474/task/42599/exe (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42599 cephadm ceph NOFD /proc/32474/task/42599/fd (opendir: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42600 cephadm ceph cwd unknown /proc/32474/task/42600/cwd (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42600 cephadm ceph rtd unknown /proc/32474/task/42600/root (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42600 cephadm ceph txt unknown /proc/32474/task/42600/exe (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42600 cephadm ceph NOFD /proc/32474/task/42600/fd (opendir: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42633 cephadm ceph cwd unknown /proc/32474/task/42633/cwd (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42633 cephadm ceph rtd unknown /proc/32474/task/42633/root (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42633 cephadm ceph txt unknown /proc/32474/task/42633/exe (readlink: Permission denied) 2024-03-16T11:08:29.669 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42633 cephadm ceph NOFD /proc/32474/task/42633/fd (opendir: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42634 cephadm ceph cwd unknown /proc/32474/task/42634/cwd (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42634 cephadm ceph rtd unknown /proc/32474/task/42634/root (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42634 cephadm ceph txt unknown /proc/32474/task/42634/exe (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42634 cephadm ceph NOFD /proc/32474/task/42634/fd (opendir: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42635 cephadm ceph cwd unknown /proc/32474/task/42635/cwd (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42635 cephadm ceph rtd unknown /proc/32474/task/42635/root (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42635 cephadm ceph txt unknown /proc/32474/task/42635/exe (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42635 cephadm ceph NOFD /proc/32474/task/42635/fd (opendir: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42636 cephadm ceph cwd unknown /proc/32474/task/42636/cwd (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42636 cephadm ceph rtd unknown /proc/32474/task/42636/root (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42636 cephadm ceph txt unknown /proc/32474/task/42636/exe (readlink: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 42636 cephadm ceph NOFD /proc/32474/task/42636/fd (opendir: Permission denied) 2024-03-16T11:08:29.670 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51646 devicehea ceph cwd unknown /proc/32474/task/51646/cwd (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51646 devicehea ceph rtd unknown /proc/32474/task/51646/root (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51646 devicehea ceph txt unknown /proc/32474/task/51646/exe (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51646 devicehea ceph NOFD /proc/32474/task/51646/fd (opendir: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51651 io_contex ceph cwd unknown /proc/32474/task/51651/cwd (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51651 io_contex ceph rtd unknown /proc/32474/task/51651/root (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51651 io_contex ceph txt unknown /proc/32474/task/51651/exe (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51651 io_contex ceph NOFD /proc/32474/task/51651/fd (opendir: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51652 io_contex ceph cwd unknown /proc/32474/task/51652/cwd (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51652 io_contex ceph rtd unknown /proc/32474/task/51652/root (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51652 io_contex ceph txt unknown /proc/32474/task/51652/exe (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51652 io_contex ceph NOFD /proc/32474/task/51652/fd (opendir: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51653 ceph_time ceph cwd unknown /proc/32474/task/51653/cwd (readlink: Permission denied) 2024-03-16T11:08:29.671 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51653 ceph_time ceph rtd unknown /proc/32474/task/51653/root (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51653 ceph_time ceph txt unknown /proc/32474/task/51653/exe (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51653 ceph_time ceph NOFD /proc/32474/task/51653/fd (opendir: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51654 ms_dispat ceph cwd unknown /proc/32474/task/51654/cwd (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51654 ms_dispat ceph rtd unknown /proc/32474/task/51654/root (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51654 ms_dispat ceph txt unknown /proc/32474/task/51654/exe (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51654 ms_dispat ceph NOFD /proc/32474/task/51654/fd (opendir: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51655 ms_local ceph cwd unknown /proc/32474/task/51655/cwd (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51655 ms_local ceph rtd unknown /proc/32474/task/51655/root (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51655 ms_local ceph txt unknown /proc/32474/task/51655/exe (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51655 ms_local ceph NOFD /proc/32474/task/51655/fd (opendir: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51656 safe_time ceph cwd unknown /proc/32474/task/51656/cwd (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51656 safe_time ceph rtd unknown /proc/32474/task/51656/root (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51656 safe_time ceph txt unknown /proc/32474/task/51656/exe (readlink: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51656 safe_time ceph NOFD /proc/32474/task/51656/fd (opendir: Permission denied) 2024-03-16T11:08:29.672 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51657 safe_time ceph cwd unknown /proc/32474/task/51657/cwd (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51657 safe_time ceph rtd unknown /proc/32474/task/51657/root (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51657 safe_time ceph txt unknown /proc/32474/task/51657/exe (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mgr 32474 51657 safe_time ceph NOFD /proc/32474/task/51657/fd (opendir: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 33619 root cwd unknown /proc/33619/cwd (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 33619 root rtd unknown /proc/33619/root (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 33619 root txt unknown /proc/33619/exe (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:systemd 33619 root NOFD /proc/33619/fd (opendir: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 33622 root cwd unknown /proc/33622/cwd (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 33622 root rtd unknown /proc/33622/root (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 33622 root txt unknown /proc/33622/exe (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:(sd-pam) 33622 root NOFD /proc/33622/fd (opendir: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35291 root cwd unknown /proc/35291/cwd (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35291 root rtd unknown /proc/35291/root (readlink: Permission denied) 2024-03-16T11:08:29.673 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35291 root txt unknown /proc/35291/exe (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35291 root NOFD /proc/35291/fd (opendir: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35315 root cwd unknown /proc/35315/cwd (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35315 root rtd unknown /proc/35315/root (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35315 root txt unknown /proc/35315/exe (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:sshd 35315 root NOFD /proc/35315/fd (opendir: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 38870 root cwd unknown /proc/38870/cwd (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 38870 root rtd unknown /proc/38870/root (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 38870 root txt unknown /proc/38870/exe (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/5 38870 root NOFD /proc/38870/fd (opendir: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 46624 root cwd unknown /proc/46624/cwd (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 46624 root rtd unknown /proc/46624/root (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 46624 root txt unknown /proc/46624/exe (readlink: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 46624 root NOFD /proc/46624/fd (opendir: Permission denied) 2024-03-16T11:08:29.674 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 46967 root cwd unknown /proc/46967/cwd (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 46967 root rtd unknown /proc/46967/root (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 46967 root txt unknown /proc/46967/exe (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 46967 root NOFD /proc/46967/fd (opendir: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 46969 root cwd unknown /proc/46969/cwd (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 46969 root rtd unknown /proc/46969/root (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 46969 root txt unknown /proc/46969/exe (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 46969 root NOFD /proc/46969/fd (opendir: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 ceph cwd unknown /proc/46971/cwd (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 ceph rtd unknown /proc/46971/root (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 ceph txt unknown /proc/46971/exe (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 ceph NOFD /proc/46971/fd (opendir: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46972 log ceph cwd unknown /proc/46971/task/46972/cwd (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46972 log ceph rtd unknown /proc/46971/task/46972/root (readlink: Permission denied) 2024-03-16T11:08:29.675 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46972 log ceph txt unknown /proc/46971/task/46972/exe (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46972 log ceph NOFD /proc/46971/task/46972/fd (opendir: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46974 msgr-work ceph cwd unknown /proc/46971/task/46974/cwd (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46974 msgr-work ceph rtd unknown /proc/46971/task/46974/root (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46974 msgr-work ceph txt unknown /proc/46971/task/46974/exe (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46974 msgr-work ceph NOFD /proc/46971/task/46974/fd (opendir: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46975 msgr-work ceph cwd unknown /proc/46971/task/46975/cwd (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46975 msgr-work ceph rtd unknown /proc/46971/task/46975/root (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46975 msgr-work ceph txt unknown /proc/46971/task/46975/exe (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46975 msgr-work ceph NOFD /proc/46971/task/46975/fd (opendir: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46976 msgr-work ceph cwd unknown /proc/46971/task/46976/cwd (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46976 msgr-work ceph rtd unknown /proc/46971/task/46976/root (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46976 msgr-work ceph txt unknown /proc/46971/task/46976/exe (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46976 msgr-work ceph NOFD /proc/46971/task/46976/fd (opendir: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46980 service ceph cwd unknown /proc/46971/task/46980/cwd (readlink: Permission denied) 2024-03-16T11:08:29.676 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46980 service ceph rtd unknown /proc/46971/task/46980/root (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46980 service ceph txt unknown /proc/46971/task/46980/exe (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46980 service ceph NOFD /proc/46971/task/46980/fd (opendir: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46981 admin_soc ceph cwd unknown /proc/46971/task/46981/cwd (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46981 admin_soc ceph rtd unknown /proc/46971/task/46981/root (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46981 admin_soc ceph txt unknown /proc/46971/task/46981/exe (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 46981 admin_soc ceph NOFD /proc/46971/task/46981/fd (opendir: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47123 signal_ha ceph cwd unknown /proc/46971/task/47123/cwd (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47123 signal_ha ceph rtd unknown /proc/46971/task/47123/root (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47123 signal_ha ceph txt unknown /proc/46971/task/47123/exe (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47123 signal_ha ceph NOFD /proc/46971/task/47123/fd (opendir: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47124 io_contex ceph cwd unknown /proc/46971/task/47124/cwd (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47124 io_contex ceph rtd unknown /proc/46971/task/47124/root (readlink: Permission denied) 2024-03-16T11:08:29.677 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47124 io_contex ceph txt unknown /proc/46971/task/47124/exe (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47124 io_contex ceph NOFD /proc/46971/task/47124/fd (opendir: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47125 io_contex ceph cwd unknown /proc/46971/task/47125/cwd (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47125 io_contex ceph rtd unknown /proc/46971/task/47125/root (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47125 io_contex ceph txt unknown /proc/46971/task/47125/exe (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47125 io_contex ceph NOFD /proc/46971/task/47125/fd (opendir: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47243 OpHistory ceph cwd unknown /proc/46971/task/47243/cwd (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47243 OpHistory ceph rtd unknown /proc/46971/task/47243/root (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47243 OpHistory ceph txt unknown /proc/46971/task/47243/exe (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47243 OpHistory ceph NOFD /proc/46971/task/47243/fd (opendir: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47244 ceph_time ceph cwd unknown /proc/46971/task/47244/cwd (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47244 ceph_time ceph rtd unknown /proc/46971/task/47244/root (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47244 ceph_time ceph txt unknown /proc/46971/task/47244/exe (readlink: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47244 ceph_time ceph NOFD /proc/46971/task/47244/fd (opendir: Permission denied) 2024-03-16T11:08:29.678 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47265 ceph-osd ceph cwd unknown /proc/46971/task/47265/cwd (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47265 ceph-osd ceph rtd unknown /proc/46971/task/47265/root (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47265 ceph-osd ceph txt unknown /proc/46971/task/47265/exe (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47265 ceph-osd ceph NOFD /proc/46971/task/47265/fd (opendir: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47272 ceph-osd ceph cwd unknown /proc/46971/task/47272/cwd (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47272 ceph-osd ceph rtd unknown /proc/46971/task/47272/root (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47272 ceph-osd ceph txt unknown /proc/46971/task/47272/exe (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47272 ceph-osd ceph NOFD /proc/46971/task/47272/fd (opendir: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47277 ceph-osd ceph cwd unknown /proc/46971/task/47277/cwd (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47277 ceph-osd ceph rtd unknown /proc/46971/task/47277/root (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47277 ceph-osd ceph txt unknown /proc/46971/task/47277/exe (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47277 ceph-osd ceph NOFD /proc/46971/task/47277/fd (opendir: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47302 ceph-osd ceph cwd unknown /proc/46971/task/47302/cwd (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47302 ceph-osd ceph rtd unknown /proc/46971/task/47302/root (readlink: Permission denied) 2024-03-16T11:08:29.679 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47302 ceph-osd ceph txt unknown /proc/46971/task/47302/exe (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47302 ceph-osd ceph NOFD /proc/46971/task/47302/fd (opendir: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47326 ceph-osd ceph cwd unknown /proc/46971/task/47326/cwd (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47326 ceph-osd ceph rtd unknown /proc/46971/task/47326/root (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47326 ceph-osd ceph txt unknown /proc/46971/task/47326/exe (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47326 ceph-osd ceph NOFD /proc/46971/task/47326/fd (opendir: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47366 ceph-osd ceph cwd unknown /proc/46971/task/47366/cwd (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47366 ceph-osd ceph rtd unknown /proc/46971/task/47366/root (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47366 ceph-osd ceph txt unknown /proc/46971/task/47366/exe (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47366 ceph-osd ceph NOFD /proc/46971/task/47366/fd (opendir: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47405 ceph-osd ceph cwd unknown /proc/46971/task/47405/cwd (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47405 ceph-osd ceph rtd unknown /proc/46971/task/47405/root (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47405 ceph-osd ceph txt unknown /proc/46971/task/47405/exe (readlink: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47405 ceph-osd ceph NOFD /proc/46971/task/47405/fd (opendir: Permission denied) 2024-03-16T11:08:29.680 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47410 ceph-osd ceph cwd unknown /proc/46971/task/47410/cwd (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47410 ceph-osd ceph rtd unknown /proc/46971/task/47410/root (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47410 ceph-osd ceph txt unknown /proc/46971/task/47410/exe (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47410 ceph-osd ceph NOFD /proc/46971/task/47410/fd (opendir: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47412 ceph-osd ceph cwd unknown /proc/46971/task/47412/cwd (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47412 ceph-osd ceph rtd unknown /proc/46971/task/47412/root (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47412 ceph-osd ceph txt unknown /proc/46971/task/47412/exe (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47412 ceph-osd ceph NOFD /proc/46971/task/47412/fd (opendir: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47413 safe_time ceph cwd unknown /proc/46971/task/47413/cwd (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47413 safe_time ceph rtd unknown /proc/46971/task/47413/root (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47413 safe_time ceph txt unknown /proc/46971/task/47413/exe (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47413 safe_time ceph NOFD /proc/46971/task/47413/fd (opendir: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47414 safe_time ceph cwd unknown /proc/46971/task/47414/cwd (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47414 safe_time ceph rtd unknown /proc/46971/task/47414/root (readlink: Permission denied) 2024-03-16T11:08:29.681 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47414 safe_time ceph txt unknown /proc/46971/task/47414/exe (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47414 safe_time ceph NOFD /proc/46971/task/47414/fd (opendir: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47415 safe_time ceph cwd unknown /proc/46971/task/47415/cwd (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47415 safe_time ceph rtd unknown /proc/46971/task/47415/root (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47415 safe_time ceph txt unknown /proc/46971/task/47415/exe (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47415 safe_time ceph NOFD /proc/46971/task/47415/fd (opendir: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47416 safe_time ceph cwd unknown /proc/46971/task/47416/cwd (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47416 safe_time ceph rtd unknown /proc/46971/task/47416/root (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47416 safe_time ceph txt unknown /proc/46971/task/47416/exe (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47416 safe_time ceph NOFD /proc/46971/task/47416/fd (opendir: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47417 fn_anonym ceph cwd unknown /proc/46971/task/47417/cwd (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47417 fn_anonym ceph rtd unknown /proc/46971/task/47417/root (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47417 fn_anonym ceph txt unknown /proc/46971/task/47417/exe (readlink: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47417 fn_anonym ceph NOFD /proc/46971/task/47417/fd (opendir: Permission denied) 2024-03-16T11:08:29.682 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47421 rocksdb:l ceph cwd unknown /proc/46971/task/47421/cwd (readlink: Permission denied) 2024-03-16T11:08:29.683 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47421 rocksdb:l ceph rtd unknown /proc/46971/task/47421/root (readlink: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47421 rocksdb:l ceph txt unknown /proc/46971/task/47421/exe (readlink: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47421 rocksdb:l ceph NOFD /proc/46971/task/47421/fd (opendir: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47422 rocksdb:l ceph cwd unknown /proc/46971/task/47422/cwd (readlink: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47422 rocksdb:l ceph rtd unknown /proc/46971/task/47422/root (readlink: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47422 rocksdb:l ceph txt unknown /proc/46971/task/47422/exe (readlink: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47422 rocksdb:l ceph NOFD /proc/46971/task/47422/fd (opendir: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47423 rocksdb:l ceph cwd unknown /proc/46971/task/47423/cwd (readlink: Permission denied) 2024-03-16T11:08:29.685 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47423 rocksdb:l ceph rtd unknown /proc/46971/task/47423/root (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47423 rocksdb:l ceph txt unknown /proc/46971/task/47423/exe (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47423 rocksdb:l ceph NOFD /proc/46971/task/47423/fd (opendir: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47424 rocksdb:h ceph cwd unknown /proc/46971/task/47424/cwd (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47424 rocksdb:h ceph rtd unknown /proc/46971/task/47424/root (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47424 rocksdb:h ceph txt unknown /proc/46971/task/47424/exe (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47424 rocksdb:h ceph NOFD /proc/46971/task/47424/fd (opendir: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47831 bstore_ai ceph cwd unknown /proc/46971/task/47831/cwd (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47831 bstore_ai ceph rtd unknown /proc/46971/task/47831/root (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47831 bstore_ai ceph txt unknown /proc/46971/task/47831/exe (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 47831 bstore_ai ceph NOFD /proc/46971/task/47831/fd (opendir: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48094 bstore_ai ceph cwd unknown /proc/46971/task/48094/cwd (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48094 bstore_ai ceph rtd unknown /proc/46971/task/48094/root (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48094 bstore_ai ceph txt unknown /proc/46971/task/48094/exe (readlink: Permission denied) 2024-03-16T11:08:29.686 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48094 bstore_ai ceph NOFD /proc/46971/task/48094/fd (opendir: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48276 ceph-osd ceph cwd unknown /proc/46971/task/48276/cwd (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48276 ceph-osd ceph rtd unknown /proc/46971/task/48276/root (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48276 ceph-osd ceph txt unknown /proc/46971/task/48276/exe (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48276 ceph-osd ceph NOFD /proc/46971/task/48276/fd (opendir: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48277 cfin ceph cwd unknown /proc/46971/task/48277/cwd (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48277 cfin ceph rtd unknown /proc/46971/task/48277/root (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48277 cfin ceph txt unknown /proc/46971/task/48277/exe (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48277 cfin ceph NOFD /proc/46971/task/48277/fd (opendir: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48278 bstore_kv ceph cwd unknown /proc/46971/task/48278/cwd (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48278 bstore_kv ceph rtd unknown /proc/46971/task/48278/root (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48278 bstore_kv ceph txt unknown /proc/46971/task/48278/exe (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48278 bstore_kv ceph NOFD /proc/46971/task/48278/fd (opendir: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48279 bstore_kv ceph cwd unknown /proc/46971/task/48279/cwd (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48279 bstore_kv ceph rtd unknown /proc/46971/task/48279/root (readlink: Permission denied) 2024-03-16T11:08:29.687 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48279 bstore_kv ceph txt unknown /proc/46971/task/48279/exe (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48279 bstore_kv ceph NOFD /proc/46971/task/48279/fd (opendir: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48280 bstore_me ceph cwd unknown /proc/46971/task/48280/cwd (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48280 bstore_me ceph rtd unknown /proc/46971/task/48280/root (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48280 bstore_me ceph txt unknown /proc/46971/task/48280/exe (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48280 bstore_me ceph NOFD /proc/46971/task/48280/fd (opendir: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48281 ms_dispat ceph cwd unknown /proc/46971/task/48281/cwd (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48281 ms_dispat ceph rtd unknown /proc/46971/task/48281/root (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48281 ms_dispat ceph txt unknown /proc/46971/task/48281/exe (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48281 ms_dispat ceph NOFD /proc/46971/task/48281/fd (opendir: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48282 ms_local ceph cwd unknown /proc/46971/task/48282/cwd (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48282 ms_local ceph rtd unknown /proc/46971/task/48282/root (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48282 ms_local ceph txt unknown /proc/46971/task/48282/exe (readlink: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48282 ms_local ceph NOFD /proc/46971/task/48282/fd (opendir: Permission denied) 2024-03-16T11:08:29.688 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48283 safe_time ceph cwd unknown /proc/46971/task/48283/cwd (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48283 safe_time ceph rtd unknown /proc/46971/task/48283/root (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48283 safe_time ceph txt unknown /proc/46971/task/48283/exe (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48283 safe_time ceph NOFD /proc/46971/task/48283/fd (opendir: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48284 safe_time ceph cwd unknown /proc/46971/task/48284/cwd (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48284 safe_time ceph rtd unknown /proc/46971/task/48284/root (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48284 safe_time ceph txt unknown /proc/46971/task/48284/exe (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48284 safe_time ceph NOFD /proc/46971/task/48284/fd (opendir: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48285 ms_dispat ceph cwd unknown /proc/46971/task/48285/cwd (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48285 ms_dispat ceph rtd unknown /proc/46971/task/48285/root (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48285 ms_dispat ceph txt unknown /proc/46971/task/48285/exe (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48285 ms_dispat ceph NOFD /proc/46971/task/48285/fd (opendir: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48286 ms_local ceph cwd unknown /proc/46971/task/48286/cwd (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48286 ms_local ceph rtd unknown /proc/46971/task/48286/root (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48286 ms_local ceph txt unknown /proc/46971/task/48286/exe (readlink: Permission denied) 2024-03-16T11:08:29.689 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48286 ms_local ceph NOFD /proc/46971/task/48286/fd (opendir: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48287 ms_dispat ceph cwd unknown /proc/46971/task/48287/cwd (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48287 ms_dispat ceph rtd unknown /proc/46971/task/48287/root (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48287 ms_dispat ceph txt unknown /proc/46971/task/48287/exe (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48287 ms_dispat ceph NOFD /proc/46971/task/48287/fd (opendir: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48288 ms_local ceph cwd unknown /proc/46971/task/48288/cwd (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48288 ms_local ceph rtd unknown /proc/46971/task/48288/root (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48288 ms_local ceph txt unknown /proc/46971/task/48288/exe (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48288 ms_local ceph NOFD /proc/46971/task/48288/fd (opendir: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48289 ms_dispat ceph cwd unknown /proc/46971/task/48289/cwd (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48289 ms_dispat ceph rtd unknown /proc/46971/task/48289/root (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48289 ms_dispat ceph txt unknown /proc/46971/task/48289/exe (readlink: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48289 ms_dispat ceph NOFD /proc/46971/task/48289/fd (opendir: Permission denied) 2024-03-16T11:08:29.690 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48290 ms_local ceph cwd unknown /proc/46971/task/48290/cwd (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48290 ms_local ceph rtd unknown /proc/46971/task/48290/root (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48290 ms_local ceph txt unknown /proc/46971/task/48290/exe (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48290 ms_local ceph NOFD /proc/46971/task/48290/fd (opendir: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48291 ms_dispat ceph cwd unknown /proc/46971/task/48291/cwd (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48291 ms_dispat ceph rtd unknown /proc/46971/task/48291/root (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48291 ms_dispat ceph txt unknown /proc/46971/task/48291/exe (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48291 ms_dispat ceph NOFD /proc/46971/task/48291/fd (opendir: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48292 ms_local ceph cwd unknown /proc/46971/task/48292/cwd (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48292 ms_local ceph rtd unknown /proc/46971/task/48292/root (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48292 ms_local ceph txt unknown /proc/46971/task/48292/exe (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48292 ms_local ceph NOFD /proc/46971/task/48292/fd (opendir: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48293 ms_dispat ceph cwd unknown /proc/46971/task/48293/cwd (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48293 ms_dispat ceph rtd unknown /proc/46971/task/48293/root (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48293 ms_dispat ceph txt unknown /proc/46971/task/48293/exe (readlink: Permission denied) 2024-03-16T11:08:29.691 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48293 ms_dispat ceph NOFD /proc/46971/task/48293/fd (opendir: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48294 ms_local ceph cwd unknown /proc/46971/task/48294/cwd (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48294 ms_local ceph rtd unknown /proc/46971/task/48294/root (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48294 ms_local ceph txt unknown /proc/46971/task/48294/exe (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48294 ms_local ceph NOFD /proc/46971/task/48294/fd (opendir: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48295 ms_dispat ceph cwd unknown /proc/46971/task/48295/cwd (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48295 ms_dispat ceph rtd unknown /proc/46971/task/48295/root (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48295 ms_dispat ceph txt unknown /proc/46971/task/48295/exe (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48295 ms_dispat ceph NOFD /proc/46971/task/48295/fd (opendir: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48296 ms_local ceph cwd unknown /proc/46971/task/48296/cwd (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48296 ms_local ceph rtd unknown /proc/46971/task/48296/root (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48296 ms_local ceph txt unknown /proc/46971/task/48296/exe (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48296 ms_local ceph NOFD /proc/46971/task/48296/fd (opendir: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48297 fn_anonym ceph cwd unknown /proc/46971/task/48297/cwd (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48297 fn_anonym ceph rtd unknown /proc/46971/task/48297/root (readlink: Permission denied) 2024-03-16T11:08:29.692 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48297 fn_anonym ceph txt unknown /proc/46971/task/48297/exe (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48297 fn_anonym ceph NOFD /proc/46971/task/48297/fd (opendir: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48298 finisher ceph cwd unknown /proc/46971/task/48298/cwd (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48298 finisher ceph rtd unknown /proc/46971/task/48298/root (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48298 finisher ceph txt unknown /proc/46971/task/48298/exe (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48298 finisher ceph NOFD /proc/46971/task/48298/fd (opendir: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48299 safe_time ceph cwd unknown /proc/46971/task/48299/cwd (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48299 safe_time ceph rtd unknown /proc/46971/task/48299/root (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48299 safe_time ceph txt unknown /proc/46971/task/48299/exe (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48299 safe_time ceph NOFD /proc/46971/task/48299/fd (opendir: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48300 safe_time ceph cwd unknown /proc/46971/task/48300/cwd (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48300 safe_time ceph rtd unknown /proc/46971/task/48300/root (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48300 safe_time ceph txt unknown /proc/46971/task/48300/exe (readlink: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48300 safe_time ceph NOFD /proc/46971/task/48300/fd (opendir: Permission denied) 2024-03-16T11:08:29.693 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48301 ceph-osd ceph cwd unknown /proc/46971/task/48301/cwd (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48301 ceph-osd ceph rtd unknown /proc/46971/task/48301/root (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48301 ceph-osd ceph txt unknown /proc/46971/task/48301/exe (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48301 ceph-osd ceph NOFD /proc/46971/task/48301/fd (opendir: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48302 osd_srv_a ceph cwd unknown /proc/46971/task/48302/cwd (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48302 osd_srv_a ceph rtd unknown /proc/46971/task/48302/root (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48302 osd_srv_a ceph txt unknown /proc/46971/task/48302/exe (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48302 osd_srv_a ceph NOFD /proc/46971/task/48302/fd (opendir: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48303 tp_osd_tp ceph cwd unknown /proc/46971/task/48303/cwd (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48303 tp_osd_tp ceph rtd unknown /proc/46971/task/48303/root (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48303 tp_osd_tp ceph txt unknown /proc/46971/task/48303/exe (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48303 tp_osd_tp ceph NOFD /proc/46971/task/48303/fd (opendir: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48304 tp_osd_tp ceph cwd unknown /proc/46971/task/48304/cwd (readlink: Permission denied) 2024-03-16T11:08:29.694 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48304 tp_osd_tp ceph rtd unknown /proc/46971/task/48304/root (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48304 tp_osd_tp ceph txt unknown /proc/46971/task/48304/exe (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48304 tp_osd_tp ceph NOFD /proc/46971/task/48304/fd (opendir: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48305 tp_osd_tp ceph cwd unknown /proc/46971/task/48305/cwd (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48305 tp_osd_tp ceph rtd unknown /proc/46971/task/48305/root (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48305 tp_osd_tp ceph txt unknown /proc/46971/task/48305/exe (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48305 tp_osd_tp ceph NOFD /proc/46971/task/48305/fd (opendir: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48306 tp_osd_tp ceph cwd unknown /proc/46971/task/48306/cwd (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48306 tp_osd_tp ceph rtd unknown /proc/46971/task/48306/root (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48306 tp_osd_tp ceph txt unknown /proc/46971/task/48306/exe (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48306 tp_osd_tp ceph NOFD /proc/46971/task/48306/fd (opendir: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48307 tp_osd_tp ceph cwd unknown /proc/46971/task/48307/cwd (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48307 tp_osd_tp ceph rtd unknown /proc/46971/task/48307/root (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48307 tp_osd_tp ceph txt unknown /proc/46971/task/48307/exe (readlink: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48307 tp_osd_tp ceph NOFD /proc/46971/task/48307/fd (opendir: Permission denied) 2024-03-16T11:08:29.695 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48308 tp_osd_tp ceph cwd unknown /proc/46971/task/48308/cwd (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48308 tp_osd_tp ceph rtd unknown /proc/46971/task/48308/root (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48308 tp_osd_tp ceph txt unknown /proc/46971/task/48308/exe (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48308 tp_osd_tp ceph NOFD /proc/46971/task/48308/fd (opendir: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48309 tp_osd_tp ceph cwd unknown /proc/46971/task/48309/cwd (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48309 tp_osd_tp ceph rtd unknown /proc/46971/task/48309/root (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48309 tp_osd_tp ceph txt unknown /proc/46971/task/48309/exe (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48309 tp_osd_tp ceph NOFD /proc/46971/task/48309/fd (opendir: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48310 tp_osd_tp ceph cwd unknown /proc/46971/task/48310/cwd (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48310 tp_osd_tp ceph rtd unknown /proc/46971/task/48310/root (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48310 tp_osd_tp ceph txt unknown /proc/46971/task/48310/exe (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48310 tp_osd_tp ceph NOFD /proc/46971/task/48310/fd (opendir: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48311 tp_osd_tp ceph cwd unknown /proc/46971/task/48311/cwd (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48311 tp_osd_tp ceph rtd unknown /proc/46971/task/48311/root (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48311 tp_osd_tp ceph txt unknown /proc/46971/task/48311/exe (readlink: Permission denied) 2024-03-16T11:08:29.696 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48311 tp_osd_tp ceph NOFD /proc/46971/task/48311/fd (opendir: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48312 tp_osd_tp ceph cwd unknown /proc/46971/task/48312/cwd (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48312 tp_osd_tp ceph rtd unknown /proc/46971/task/48312/root (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48312 tp_osd_tp ceph txt unknown /proc/46971/task/48312/exe (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48312 tp_osd_tp ceph NOFD /proc/46971/task/48312/fd (opendir: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48313 tp_osd_tp ceph cwd unknown /proc/46971/task/48313/cwd (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48313 tp_osd_tp ceph rtd unknown /proc/46971/task/48313/root (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48313 tp_osd_tp ceph txt unknown /proc/46971/task/48313/exe (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48313 tp_osd_tp ceph NOFD /proc/46971/task/48313/fd (opendir: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48314 tp_osd_tp ceph cwd unknown /proc/46971/task/48314/cwd (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48314 tp_osd_tp ceph rtd unknown /proc/46971/task/48314/root (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48314 tp_osd_tp ceph txt unknown /proc/46971/task/48314/exe (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48314 tp_osd_tp ceph NOFD /proc/46971/task/48314/fd (opendir: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48315 tp_osd_tp ceph cwd unknown /proc/46971/task/48315/cwd (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48315 tp_osd_tp ceph rtd unknown /proc/46971/task/48315/root (readlink: Permission denied) 2024-03-16T11:08:29.697 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48315 tp_osd_tp ceph txt unknown /proc/46971/task/48315/exe (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48315 tp_osd_tp ceph NOFD /proc/46971/task/48315/fd (opendir: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48316 tp_osd_tp ceph cwd unknown /proc/46971/task/48316/cwd (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48316 tp_osd_tp ceph rtd unknown /proc/46971/task/48316/root (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48316 tp_osd_tp ceph txt unknown /proc/46971/task/48316/exe (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48316 tp_osd_tp ceph NOFD /proc/46971/task/48316/fd (opendir: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48317 tp_osd_tp ceph cwd unknown /proc/46971/task/48317/cwd (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48317 tp_osd_tp ceph rtd unknown /proc/46971/task/48317/root (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48317 tp_osd_tp ceph txt unknown /proc/46971/task/48317/exe (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48317 tp_osd_tp ceph NOFD /proc/46971/task/48317/fd (opendir: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48318 tp_osd_tp ceph cwd unknown /proc/46971/task/48318/cwd (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48318 tp_osd_tp ceph rtd unknown /proc/46971/task/48318/root (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48318 tp_osd_tp ceph txt unknown /proc/46971/task/48318/exe (readlink: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48318 tp_osd_tp ceph NOFD /proc/46971/task/48318/fd (opendir: Permission denied) 2024-03-16T11:08:29.698 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48319 osd_srv_h ceph cwd unknown /proc/46971/task/48319/cwd (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48319 osd_srv_h ceph rtd unknown /proc/46971/task/48319/root (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48319 osd_srv_h ceph txt unknown /proc/46971/task/48319/exe (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 46971 48319 osd_srv_h ceph NOFD /proc/46971/task/48319/fd (opendir: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 48332 root cwd unknown /proc/48332/cwd (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 48332 root rtd unknown /proc/48332/root (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 48332 root txt unknown /proc/48332/exe (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 48332 root NOFD /proc/48332/fd (opendir: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 48356 root cwd unknown /proc/48356/cwd (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 48356 root rtd unknown /proc/48356/root (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 48356 root txt unknown /proc/48356/exe (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 48356 root NOFD /proc/48356/fd (opendir: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 53953 root cwd unknown /proc/53953/cwd (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 53953 root rtd unknown /proc/53953/root (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 53953 root txt unknown /proc/53953/exe (readlink: Permission denied) 2024-03-16T11:08:29.699 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 53953 root NOFD /proc/53953/fd (opendir: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 53955 root cwd unknown /proc/53955/cwd (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 53955 root rtd unknown /proc/53955/root (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 53955 root txt unknown /proc/53955/exe (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 53955 root NOFD /proc/53955/fd (opendir: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 ceph cwd unknown /proc/53957/cwd (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 ceph rtd unknown /proc/53957/root (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 ceph txt unknown /proc/53957/exe (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 ceph NOFD /proc/53957/fd (opendir: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53958 log ceph cwd unknown /proc/53957/task/53958/cwd (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53958 log ceph rtd unknown /proc/53957/task/53958/root (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53958 log ceph txt unknown /proc/53957/task/53958/exe (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53958 log ceph NOFD /proc/53957/task/53958/fd (opendir: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53960 msgr-work ceph cwd unknown /proc/53957/task/53960/cwd (readlink: Permission denied) 2024-03-16T11:08:29.700 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53960 msgr-work ceph rtd unknown /proc/53957/task/53960/root (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53960 msgr-work ceph txt unknown /proc/53957/task/53960/exe (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53960 msgr-work ceph NOFD /proc/53957/task/53960/fd (opendir: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53961 msgr-work ceph cwd unknown /proc/53957/task/53961/cwd (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53961 msgr-work ceph rtd unknown /proc/53957/task/53961/root (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53961 msgr-work ceph txt unknown /proc/53957/task/53961/exe (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53961 msgr-work ceph NOFD /proc/53957/task/53961/fd (opendir: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53962 msgr-work ceph cwd unknown /proc/53957/task/53962/cwd (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53962 msgr-work ceph rtd unknown /proc/53957/task/53962/root (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53962 msgr-work ceph txt unknown /proc/53957/task/53962/exe (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53962 msgr-work ceph NOFD /proc/53957/task/53962/fd (opendir: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53966 service ceph cwd unknown /proc/53957/task/53966/cwd (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53966 service ceph rtd unknown /proc/53957/task/53966/root (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53966 service ceph txt unknown /proc/53957/task/53966/exe (readlink: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53966 service ceph NOFD /proc/53957/task/53966/fd (opendir: Permission denied) 2024-03-16T11:08:29.701 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53967 admin_soc ceph cwd unknown /proc/53957/task/53967/cwd (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53967 admin_soc ceph rtd unknown /proc/53957/task/53967/root (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53967 admin_soc ceph txt unknown /proc/53957/task/53967/exe (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 53967 admin_soc ceph NOFD /proc/53957/task/53967/fd (opendir: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54109 signal_ha ceph cwd unknown /proc/53957/task/54109/cwd (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54109 signal_ha ceph rtd unknown /proc/53957/task/54109/root (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54109 signal_ha ceph txt unknown /proc/53957/task/54109/exe (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54109 signal_ha ceph NOFD /proc/53957/task/54109/fd (opendir: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54110 io_contex ceph cwd unknown /proc/53957/task/54110/cwd (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54110 io_contex ceph rtd unknown /proc/53957/task/54110/root (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54110 io_contex ceph txt unknown /proc/53957/task/54110/exe (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54110 io_contex ceph NOFD /proc/53957/task/54110/fd (opendir: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54111 io_contex ceph cwd unknown /proc/53957/task/54111/cwd (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54111 io_contex ceph rtd unknown /proc/53957/task/54111/root (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54111 io_contex ceph txt unknown /proc/53957/task/54111/exe (readlink: Permission denied) 2024-03-16T11:08:29.702 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54111 io_contex ceph NOFD /proc/53957/task/54111/fd (opendir: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54228 OpHistory ceph cwd unknown /proc/53957/task/54228/cwd (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54228 OpHistory ceph rtd unknown /proc/53957/task/54228/root (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54228 OpHistory ceph txt unknown /proc/53957/task/54228/exe (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54228 OpHistory ceph NOFD /proc/53957/task/54228/fd (opendir: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54229 ceph_time ceph cwd unknown /proc/53957/task/54229/cwd (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54229 ceph_time ceph rtd unknown /proc/53957/task/54229/root (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54229 ceph_time ceph txt unknown /proc/53957/task/54229/exe (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54229 ceph_time ceph NOFD /proc/53957/task/54229/fd (opendir: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54263 ceph-osd ceph cwd unknown /proc/53957/task/54263/cwd (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54263 ceph-osd ceph rtd unknown /proc/53957/task/54263/root (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54263 ceph-osd ceph txt unknown /proc/53957/task/54263/exe (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54263 ceph-osd ceph NOFD /proc/53957/task/54263/fd (opendir: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54268 ceph-osd ceph cwd unknown /proc/53957/task/54268/cwd (readlink: Permission denied) 2024-03-16T11:08:29.703 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54268 ceph-osd ceph rtd unknown /proc/53957/task/54268/root (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54268 ceph-osd ceph txt unknown /proc/53957/task/54268/exe (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54268 ceph-osd ceph NOFD /proc/53957/task/54268/fd (opendir: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54287 ceph-osd ceph cwd unknown /proc/53957/task/54287/cwd (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54287 ceph-osd ceph rtd unknown /proc/53957/task/54287/root (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54287 ceph-osd ceph txt unknown /proc/53957/task/54287/exe (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54287 ceph-osd ceph NOFD /proc/53957/task/54287/fd (opendir: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54317 ceph-osd ceph cwd unknown /proc/53957/task/54317/cwd (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54317 ceph-osd ceph rtd unknown /proc/53957/task/54317/root (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54317 ceph-osd ceph txt unknown /proc/53957/task/54317/exe (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54317 ceph-osd ceph NOFD /proc/53957/task/54317/fd (opendir: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54340 ceph-osd ceph cwd unknown /proc/53957/task/54340/cwd (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54340 ceph-osd ceph rtd unknown /proc/53957/task/54340/root (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54340 ceph-osd ceph txt unknown /proc/53957/task/54340/exe (readlink: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54340 ceph-osd ceph NOFD /proc/53957/task/54340/fd (opendir: Permission denied) 2024-03-16T11:08:29.704 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54349 ceph-osd ceph cwd unknown /proc/53957/task/54349/cwd (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54349 ceph-osd ceph rtd unknown /proc/53957/task/54349/root (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54349 ceph-osd ceph txt unknown /proc/53957/task/54349/exe (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54349 ceph-osd ceph NOFD /proc/53957/task/54349/fd (opendir: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54386 ceph-osd ceph cwd unknown /proc/53957/task/54386/cwd (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54386 ceph-osd ceph rtd unknown /proc/53957/task/54386/root (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54386 ceph-osd ceph txt unknown /proc/53957/task/54386/exe (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54386 ceph-osd ceph NOFD /proc/53957/task/54386/fd (opendir: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54414 ceph-osd ceph cwd unknown /proc/53957/task/54414/cwd (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54414 ceph-osd ceph rtd unknown /proc/53957/task/54414/root (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54414 ceph-osd ceph txt unknown /proc/53957/task/54414/exe (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54414 ceph-osd ceph NOFD /proc/53957/task/54414/fd (opendir: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54415 ceph-osd ceph cwd unknown /proc/53957/task/54415/cwd (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54415 ceph-osd ceph rtd unknown /proc/53957/task/54415/root (readlink: Permission denied) 2024-03-16T11:08:29.705 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54415 ceph-osd ceph txt unknown /proc/53957/task/54415/exe (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54415 ceph-osd ceph NOFD /proc/53957/task/54415/fd (opendir: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54416 safe_time ceph cwd unknown /proc/53957/task/54416/cwd (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54416 safe_time ceph rtd unknown /proc/53957/task/54416/root (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54416 safe_time ceph txt unknown /proc/53957/task/54416/exe (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54416 safe_time ceph NOFD /proc/53957/task/54416/fd (opendir: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54417 safe_time ceph cwd unknown /proc/53957/task/54417/cwd (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54417 safe_time ceph rtd unknown /proc/53957/task/54417/root (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54417 safe_time ceph txt unknown /proc/53957/task/54417/exe (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54417 safe_time ceph NOFD /proc/53957/task/54417/fd (opendir: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54418 safe_time ceph cwd unknown /proc/53957/task/54418/cwd (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54418 safe_time ceph rtd unknown /proc/53957/task/54418/root (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54418 safe_time ceph txt unknown /proc/53957/task/54418/exe (readlink: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54418 safe_time ceph NOFD /proc/53957/task/54418/fd (opendir: Permission denied) 2024-03-16T11:08:29.706 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54420 safe_time ceph cwd unknown /proc/53957/task/54420/cwd (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54420 safe_time ceph rtd unknown /proc/53957/task/54420/root (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54420 safe_time ceph txt unknown /proc/53957/task/54420/exe (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54420 safe_time ceph NOFD /proc/53957/task/54420/fd (opendir: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54421 fn_anonym ceph cwd unknown /proc/53957/task/54421/cwd (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54421 fn_anonym ceph rtd unknown /proc/53957/task/54421/root (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54421 fn_anonym ceph txt unknown /proc/53957/task/54421/exe (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54421 fn_anonym ceph NOFD /proc/53957/task/54421/fd (opendir: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54424 rocksdb:l ceph cwd unknown /proc/53957/task/54424/cwd (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54424 rocksdb:l ceph rtd unknown /proc/53957/task/54424/root (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54424 rocksdb:l ceph txt unknown /proc/53957/task/54424/exe (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54424 rocksdb:l ceph NOFD /proc/53957/task/54424/fd (opendir: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54425 rocksdb:l ceph cwd unknown /proc/53957/task/54425/cwd (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54425 rocksdb:l ceph rtd unknown /proc/53957/task/54425/root (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54425 rocksdb:l ceph txt unknown /proc/53957/task/54425/exe (readlink: Permission denied) 2024-03-16T11:08:29.707 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54425 rocksdb:l ceph NOFD /proc/53957/task/54425/fd (opendir: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54426 rocksdb:l ceph cwd unknown /proc/53957/task/54426/cwd (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54426 rocksdb:l ceph rtd unknown /proc/53957/task/54426/root (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54426 rocksdb:l ceph txt unknown /proc/53957/task/54426/exe (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54426 rocksdb:l ceph NOFD /proc/53957/task/54426/fd (opendir: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54427 rocksdb:h ceph cwd unknown /proc/53957/task/54427/cwd (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54427 rocksdb:h ceph rtd unknown /proc/53957/task/54427/root (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54427 rocksdb:h ceph txt unknown /proc/53957/task/54427/exe (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54427 rocksdb:h ceph NOFD /proc/53957/task/54427/fd (opendir: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54866 bstore_ai ceph cwd unknown /proc/53957/task/54866/cwd (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54866 bstore_ai ceph rtd unknown /proc/53957/task/54866/root (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54866 bstore_ai ceph txt unknown /proc/53957/task/54866/exe (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 54866 bstore_ai ceph NOFD /proc/53957/task/54866/fd (opendir: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55129 bstore_ai ceph cwd unknown /proc/53957/task/55129/cwd (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55129 bstore_ai ceph rtd unknown /proc/53957/task/55129/root (readlink: Permission denied) 2024-03-16T11:08:29.708 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55129 bstore_ai ceph txt unknown /proc/53957/task/55129/exe (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55129 bstore_ai ceph NOFD /proc/53957/task/55129/fd (opendir: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55311 ceph-osd ceph cwd unknown /proc/53957/task/55311/cwd (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55311 ceph-osd ceph rtd unknown /proc/53957/task/55311/root (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55311 ceph-osd ceph txt unknown /proc/53957/task/55311/exe (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55311 ceph-osd ceph NOFD /proc/53957/task/55311/fd (opendir: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55312 cfin ceph cwd unknown /proc/53957/task/55312/cwd (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55312 cfin ceph rtd unknown /proc/53957/task/55312/root (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55312 cfin ceph txt unknown /proc/53957/task/55312/exe (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55312 cfin ceph NOFD /proc/53957/task/55312/fd (opendir: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55313 bstore_kv ceph cwd unknown /proc/53957/task/55313/cwd (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55313 bstore_kv ceph rtd unknown /proc/53957/task/55313/root (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55313 bstore_kv ceph txt unknown /proc/53957/task/55313/exe (readlink: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55313 bstore_kv ceph NOFD /proc/53957/task/55313/fd (opendir: Permission denied) 2024-03-16T11:08:29.709 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55314 bstore_kv ceph cwd unknown /proc/53957/task/55314/cwd (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55314 bstore_kv ceph rtd unknown /proc/53957/task/55314/root (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55314 bstore_kv ceph txt unknown /proc/53957/task/55314/exe (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55314 bstore_kv ceph NOFD /proc/53957/task/55314/fd (opendir: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55315 bstore_me ceph cwd unknown /proc/53957/task/55315/cwd (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55315 bstore_me ceph rtd unknown /proc/53957/task/55315/root (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55315 bstore_me ceph txt unknown /proc/53957/task/55315/exe (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55315 bstore_me ceph NOFD /proc/53957/task/55315/fd (opendir: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55316 ms_dispat ceph cwd unknown /proc/53957/task/55316/cwd (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55316 ms_dispat ceph rtd unknown /proc/53957/task/55316/root (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55316 ms_dispat ceph txt unknown /proc/53957/task/55316/exe (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55316 ms_dispat ceph NOFD /proc/53957/task/55316/fd (opendir: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55317 ms_local ceph cwd unknown /proc/53957/task/55317/cwd (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55317 ms_local ceph rtd unknown /proc/53957/task/55317/root (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55317 ms_local ceph txt unknown /proc/53957/task/55317/exe (readlink: Permission denied) 2024-03-16T11:08:29.710 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55317 ms_local ceph NOFD /proc/53957/task/55317/fd (opendir: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55318 safe_time ceph cwd unknown /proc/53957/task/55318/cwd (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55318 safe_time ceph rtd unknown /proc/53957/task/55318/root (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55318 safe_time ceph txt unknown /proc/53957/task/55318/exe (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55318 safe_time ceph NOFD /proc/53957/task/55318/fd (opendir: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55319 safe_time ceph cwd unknown /proc/53957/task/55319/cwd (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55319 safe_time ceph rtd unknown /proc/53957/task/55319/root (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55319 safe_time ceph txt unknown /proc/53957/task/55319/exe (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55319 safe_time ceph NOFD /proc/53957/task/55319/fd (opendir: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55320 ms_dispat ceph cwd unknown /proc/53957/task/55320/cwd (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55320 ms_dispat ceph rtd unknown /proc/53957/task/55320/root (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55320 ms_dispat ceph txt unknown /proc/53957/task/55320/exe (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55320 ms_dispat ceph NOFD /proc/53957/task/55320/fd (opendir: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55321 ms_local ceph cwd unknown /proc/53957/task/55321/cwd (readlink: Permission denied) 2024-03-16T11:08:29.711 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55321 ms_local ceph rtd unknown /proc/53957/task/55321/root (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55321 ms_local ceph txt unknown /proc/53957/task/55321/exe (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55321 ms_local ceph NOFD /proc/53957/task/55321/fd (opendir: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55322 ms_dispat ceph cwd unknown /proc/53957/task/55322/cwd (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55322 ms_dispat ceph rtd unknown /proc/53957/task/55322/root (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55322 ms_dispat ceph txt unknown /proc/53957/task/55322/exe (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55322 ms_dispat ceph NOFD /proc/53957/task/55322/fd (opendir: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55323 ms_local ceph cwd unknown /proc/53957/task/55323/cwd (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55323 ms_local ceph rtd unknown /proc/53957/task/55323/root (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55323 ms_local ceph txt unknown /proc/53957/task/55323/exe (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55323 ms_local ceph NOFD /proc/53957/task/55323/fd (opendir: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55324 ms_dispat ceph cwd unknown /proc/53957/task/55324/cwd (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55324 ms_dispat ceph rtd unknown /proc/53957/task/55324/root (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55324 ms_dispat ceph txt unknown /proc/53957/task/55324/exe (readlink: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55324 ms_dispat ceph NOFD /proc/53957/task/55324/fd (opendir: Permission denied) 2024-03-16T11:08:29.712 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55325 ms_local ceph cwd unknown /proc/53957/task/55325/cwd (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55325 ms_local ceph rtd unknown /proc/53957/task/55325/root (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55325 ms_local ceph txt unknown /proc/53957/task/55325/exe (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55325 ms_local ceph NOFD /proc/53957/task/55325/fd (opendir: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55326 ms_dispat ceph cwd unknown /proc/53957/task/55326/cwd (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55326 ms_dispat ceph rtd unknown /proc/53957/task/55326/root (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55326 ms_dispat ceph txt unknown /proc/53957/task/55326/exe (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55326 ms_dispat ceph NOFD /proc/53957/task/55326/fd (opendir: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55327 ms_local ceph cwd unknown /proc/53957/task/55327/cwd (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55327 ms_local ceph rtd unknown /proc/53957/task/55327/root (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55327 ms_local ceph txt unknown /proc/53957/task/55327/exe (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55327 ms_local ceph NOFD /proc/53957/task/55327/fd (opendir: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55328 ms_dispat ceph cwd unknown /proc/53957/task/55328/cwd (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55328 ms_dispat ceph rtd unknown /proc/53957/task/55328/root (readlink: Permission denied) 2024-03-16T11:08:29.713 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55328 ms_dispat ceph txt unknown /proc/53957/task/55328/exe (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55328 ms_dispat ceph NOFD /proc/53957/task/55328/fd (opendir: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55329 ms_local ceph cwd unknown /proc/53957/task/55329/cwd (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55329 ms_local ceph rtd unknown /proc/53957/task/55329/root (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55329 ms_local ceph txt unknown /proc/53957/task/55329/exe (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55329 ms_local ceph NOFD /proc/53957/task/55329/fd (opendir: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55330 ms_dispat ceph cwd unknown /proc/53957/task/55330/cwd (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55330 ms_dispat ceph rtd unknown /proc/53957/task/55330/root (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55330 ms_dispat ceph txt unknown /proc/53957/task/55330/exe (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55330 ms_dispat ceph NOFD /proc/53957/task/55330/fd (opendir: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55331 ms_local ceph cwd unknown /proc/53957/task/55331/cwd (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55331 ms_local ceph rtd unknown /proc/53957/task/55331/root (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55331 ms_local ceph txt unknown /proc/53957/task/55331/exe (readlink: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55331 ms_local ceph NOFD /proc/53957/task/55331/fd (opendir: Permission denied) 2024-03-16T11:08:29.714 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55332 fn_anonym ceph cwd unknown /proc/53957/task/55332/cwd (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55332 fn_anonym ceph rtd unknown /proc/53957/task/55332/root (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55332 fn_anonym ceph txt unknown /proc/53957/task/55332/exe (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55332 fn_anonym ceph NOFD /proc/53957/task/55332/fd (opendir: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55333 finisher ceph cwd unknown /proc/53957/task/55333/cwd (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55333 finisher ceph rtd unknown /proc/53957/task/55333/root (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55333 finisher ceph txt unknown /proc/53957/task/55333/exe (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55333 finisher ceph NOFD /proc/53957/task/55333/fd (opendir: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55334 safe_time ceph cwd unknown /proc/53957/task/55334/cwd (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55334 safe_time ceph rtd unknown /proc/53957/task/55334/root (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55334 safe_time ceph txt unknown /proc/53957/task/55334/exe (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55334 safe_time ceph NOFD /proc/53957/task/55334/fd (opendir: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55335 safe_time ceph cwd unknown /proc/53957/task/55335/cwd (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55335 safe_time ceph rtd unknown /proc/53957/task/55335/root (readlink: Permission denied) 2024-03-16T11:08:29.715 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55335 safe_time ceph txt unknown /proc/53957/task/55335/exe (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55335 safe_time ceph NOFD /proc/53957/task/55335/fd (opendir: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55336 ceph-osd ceph cwd unknown /proc/53957/task/55336/cwd (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55336 ceph-osd ceph rtd unknown /proc/53957/task/55336/root (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55336 ceph-osd ceph txt unknown /proc/53957/task/55336/exe (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55336 ceph-osd ceph NOFD /proc/53957/task/55336/fd (opendir: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55337 osd_srv_a ceph cwd unknown /proc/53957/task/55337/cwd (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55337 osd_srv_a ceph rtd unknown /proc/53957/task/55337/root (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55337 osd_srv_a ceph txt unknown /proc/53957/task/55337/exe (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55337 osd_srv_a ceph NOFD /proc/53957/task/55337/fd (opendir: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55338 tp_osd_tp ceph cwd unknown /proc/53957/task/55338/cwd (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55338 tp_osd_tp ceph rtd unknown /proc/53957/task/55338/root (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55338 tp_osd_tp ceph txt unknown /proc/53957/task/55338/exe (readlink: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55338 tp_osd_tp ceph NOFD /proc/53957/task/55338/fd (opendir: Permission denied) 2024-03-16T11:08:29.716 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55339 tp_osd_tp ceph cwd unknown /proc/53957/task/55339/cwd (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55339 tp_osd_tp ceph rtd unknown /proc/53957/task/55339/root (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55339 tp_osd_tp ceph txt unknown /proc/53957/task/55339/exe (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55339 tp_osd_tp ceph NOFD /proc/53957/task/55339/fd (opendir: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55340 tp_osd_tp ceph cwd unknown /proc/53957/task/55340/cwd (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55340 tp_osd_tp ceph rtd unknown /proc/53957/task/55340/root (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55340 tp_osd_tp ceph txt unknown /proc/53957/task/55340/exe (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55340 tp_osd_tp ceph NOFD /proc/53957/task/55340/fd (opendir: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55341 tp_osd_tp ceph cwd unknown /proc/53957/task/55341/cwd (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55341 tp_osd_tp ceph rtd unknown /proc/53957/task/55341/root (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55341 tp_osd_tp ceph txt unknown /proc/53957/task/55341/exe (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55341 tp_osd_tp ceph NOFD /proc/53957/task/55341/fd (opendir: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55342 tp_osd_tp ceph cwd unknown /proc/53957/task/55342/cwd (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55342 tp_osd_tp ceph rtd unknown /proc/53957/task/55342/root (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55342 tp_osd_tp ceph txt unknown /proc/53957/task/55342/exe (readlink: Permission denied) 2024-03-16T11:08:29.717 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55342 tp_osd_tp ceph NOFD /proc/53957/task/55342/fd (opendir: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55343 tp_osd_tp ceph cwd unknown /proc/53957/task/55343/cwd (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55343 tp_osd_tp ceph rtd unknown /proc/53957/task/55343/root (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55343 tp_osd_tp ceph txt unknown /proc/53957/task/55343/exe (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55343 tp_osd_tp ceph NOFD /proc/53957/task/55343/fd (opendir: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55344 tp_osd_tp ceph cwd unknown /proc/53957/task/55344/cwd (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55344 tp_osd_tp ceph rtd unknown /proc/53957/task/55344/root (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55344 tp_osd_tp ceph txt unknown /proc/53957/task/55344/exe (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55344 tp_osd_tp ceph NOFD /proc/53957/task/55344/fd (opendir: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55345 tp_osd_tp ceph cwd unknown /proc/53957/task/55345/cwd (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55345 tp_osd_tp ceph rtd unknown /proc/53957/task/55345/root (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55345 tp_osd_tp ceph txt unknown /proc/53957/task/55345/exe (readlink: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55345 tp_osd_tp ceph NOFD /proc/53957/task/55345/fd (opendir: Permission denied) 2024-03-16T11:08:29.718 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55346 tp_osd_tp ceph cwd unknown /proc/53957/task/55346/cwd (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55346 tp_osd_tp ceph rtd unknown /proc/53957/task/55346/root (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55346 tp_osd_tp ceph txt unknown /proc/53957/task/55346/exe (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55346 tp_osd_tp ceph NOFD /proc/53957/task/55346/fd (opendir: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55347 tp_osd_tp ceph cwd unknown /proc/53957/task/55347/cwd (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55347 tp_osd_tp ceph rtd unknown /proc/53957/task/55347/root (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55347 tp_osd_tp ceph txt unknown /proc/53957/task/55347/exe (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55347 tp_osd_tp ceph NOFD /proc/53957/task/55347/fd (opendir: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55348 tp_osd_tp ceph cwd unknown /proc/53957/task/55348/cwd (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55348 tp_osd_tp ceph rtd unknown /proc/53957/task/55348/root (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55348 tp_osd_tp ceph txt unknown /proc/53957/task/55348/exe (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55348 tp_osd_tp ceph NOFD /proc/53957/task/55348/fd (opendir: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55349 tp_osd_tp ceph cwd unknown /proc/53957/task/55349/cwd (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55349 tp_osd_tp ceph rtd unknown /proc/53957/task/55349/root (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55349 tp_osd_tp ceph txt unknown /proc/53957/task/55349/exe (readlink: Permission denied) 2024-03-16T11:08:29.719 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55349 tp_osd_tp ceph NOFD /proc/53957/task/55349/fd (opendir: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55350 tp_osd_tp ceph cwd unknown /proc/53957/task/55350/cwd (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55350 tp_osd_tp ceph rtd unknown /proc/53957/task/55350/root (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55350 tp_osd_tp ceph txt unknown /proc/53957/task/55350/exe (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55350 tp_osd_tp ceph NOFD /proc/53957/task/55350/fd (opendir: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55351 tp_osd_tp ceph cwd unknown /proc/53957/task/55351/cwd (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55351 tp_osd_tp ceph rtd unknown /proc/53957/task/55351/root (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55351 tp_osd_tp ceph txt unknown /proc/53957/task/55351/exe (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55351 tp_osd_tp ceph NOFD /proc/53957/task/55351/fd (opendir: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55352 tp_osd_tp ceph cwd unknown /proc/53957/task/55352/cwd (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55352 tp_osd_tp ceph rtd unknown /proc/53957/task/55352/root (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55352 tp_osd_tp ceph txt unknown /proc/53957/task/55352/exe (readlink: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55352 tp_osd_tp ceph NOFD /proc/53957/task/55352/fd (opendir: Permission denied) 2024-03-16T11:08:29.720 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55353 tp_osd_tp ceph cwd unknown /proc/53957/task/55353/cwd (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55353 tp_osd_tp ceph rtd unknown /proc/53957/task/55353/root (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55353 tp_osd_tp ceph txt unknown /proc/53957/task/55353/exe (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55353 tp_osd_tp ceph NOFD /proc/53957/task/55353/fd (opendir: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55354 osd_srv_h ceph cwd unknown /proc/53957/task/55354/cwd (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55354 osd_srv_h ceph rtd unknown /proc/53957/task/55354/root (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55354 osd_srv_h ceph txt unknown /proc/53957/task/55354/exe (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 53957 55354 osd_srv_h ceph NOFD /proc/53957/task/55354/fd (opendir: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 55370 root cwd unknown /proc/55370/cwd (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 55370 root rtd unknown /proc/55370/root (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 55370 root txt unknown /proc/55370/exe (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 55370 root NOFD /proc/55370/fd (opendir: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 55394 root cwd unknown /proc/55394/cwd (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 55394 root rtd unknown /proc/55394/root (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 55394 root txt unknown /proc/55394/exe (readlink: Permission denied) 2024-03-16T11:08:29.721 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 55394 root NOFD /proc/55394/fd (opendir: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 55399 root cwd unknown /proc/55399/cwd (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 55399 root rtd unknown /proc/55399/root (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 55399 root txt unknown /proc/55399/exe (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 55399 root NOFD /proc/55399/fd (opendir: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 58029 root cwd unknown /proc/58029/cwd (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 58029 root rtd unknown /proc/58029/root (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 58029 root txt unknown /proc/58029/exe (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 58029 root NOFD /proc/58029/fd (opendir: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 60964 root cwd unknown /proc/60964/cwd (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 60964 root rtd unknown /proc/60964/root (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 60964 root txt unknown /proc/60964/exe (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 60964 root NOFD /proc/60964/fd (opendir: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 60966 root cwd unknown /proc/60966/cwd (readlink: Permission denied) 2024-03-16T11:08:29.722 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 60966 root rtd unknown /proc/60966/root (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 60966 root txt unknown /proc/60966/exe (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 60966 root NOFD /proc/60966/fd (opendir: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 ceph cwd unknown /proc/60968/cwd (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 ceph rtd unknown /proc/60968/root (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 ceph txt unknown /proc/60968/exe (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 ceph NOFD /proc/60968/fd (opendir: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60969 log ceph cwd unknown /proc/60968/task/60969/cwd (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60969 log ceph rtd unknown /proc/60968/task/60969/root (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60969 log ceph txt unknown /proc/60968/task/60969/exe (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60969 log ceph NOFD /proc/60968/task/60969/fd (opendir: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60971 msgr-work ceph cwd unknown /proc/60968/task/60971/cwd (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60971 msgr-work ceph rtd unknown /proc/60968/task/60971/root (readlink: Permission denied) 2024-03-16T11:08:29.723 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60971 msgr-work ceph txt unknown /proc/60968/task/60971/exe (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60971 msgr-work ceph NOFD /proc/60968/task/60971/fd (opendir: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60972 msgr-work ceph cwd unknown /proc/60968/task/60972/cwd (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60972 msgr-work ceph rtd unknown /proc/60968/task/60972/root (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60972 msgr-work ceph txt unknown /proc/60968/task/60972/exe (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60972 msgr-work ceph NOFD /proc/60968/task/60972/fd (opendir: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60973 msgr-work ceph cwd unknown /proc/60968/task/60973/cwd (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60973 msgr-work ceph rtd unknown /proc/60968/task/60973/root (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60973 msgr-work ceph txt unknown /proc/60968/task/60973/exe (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60973 msgr-work ceph NOFD /proc/60968/task/60973/fd (opendir: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60977 service ceph cwd unknown /proc/60968/task/60977/cwd (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60977 service ceph rtd unknown /proc/60968/task/60977/root (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60977 service ceph txt unknown /proc/60968/task/60977/exe (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60977 service ceph NOFD /proc/60968/task/60977/fd (opendir: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60978 admin_soc ceph cwd unknown /proc/60968/task/60978/cwd (readlink: Permission denied) 2024-03-16T11:08:29.724 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60978 admin_soc ceph rtd unknown /proc/60968/task/60978/root (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60978 admin_soc ceph txt unknown /proc/60968/task/60978/exe (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 60978 admin_soc ceph NOFD /proc/60968/task/60978/fd (opendir: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61120 signal_ha ceph cwd unknown /proc/60968/task/61120/cwd (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61120 signal_ha ceph rtd unknown /proc/60968/task/61120/root (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61120 signal_ha ceph txt unknown /proc/60968/task/61120/exe (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61120 signal_ha ceph NOFD /proc/60968/task/61120/fd (opendir: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61121 io_contex ceph cwd unknown /proc/60968/task/61121/cwd (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61121 io_contex ceph rtd unknown /proc/60968/task/61121/root (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61121 io_contex ceph txt unknown /proc/60968/task/61121/exe (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61121 io_contex ceph NOFD /proc/60968/task/61121/fd (opendir: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61122 io_contex ceph cwd unknown /proc/60968/task/61122/cwd (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61122 io_contex ceph rtd unknown /proc/60968/task/61122/root (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61122 io_contex ceph txt unknown /proc/60968/task/61122/exe (readlink: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61122 io_contex ceph NOFD /proc/60968/task/61122/fd (opendir: Permission denied) 2024-03-16T11:08:29.725 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61238 OpHistory ceph cwd unknown /proc/60968/task/61238/cwd (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61238 OpHistory ceph rtd unknown /proc/60968/task/61238/root (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61238 OpHistory ceph txt unknown /proc/60968/task/61238/exe (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61238 OpHistory ceph NOFD /proc/60968/task/61238/fd (opendir: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61239 ceph_time ceph cwd unknown /proc/60968/task/61239/cwd (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61239 ceph_time ceph rtd unknown /proc/60968/task/61239/root (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61239 ceph_time ceph txt unknown /proc/60968/task/61239/exe (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61239 ceph_time ceph NOFD /proc/60968/task/61239/fd (opendir: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61392 ceph-osd ceph cwd unknown /proc/60968/task/61392/cwd (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61392 ceph-osd ceph rtd unknown /proc/60968/task/61392/root (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61392 ceph-osd ceph txt unknown /proc/60968/task/61392/exe (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61392 ceph-osd ceph NOFD /proc/60968/task/61392/fd (opendir: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61393 safe_time ceph cwd unknown /proc/60968/task/61393/cwd (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61393 safe_time ceph rtd unknown /proc/60968/task/61393/root (readlink: Permission denied) 2024-03-16T11:08:29.726 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61393 safe_time ceph txt unknown /proc/60968/task/61393/exe (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61393 safe_time ceph NOFD /proc/60968/task/61393/fd (opendir: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61394 safe_time ceph cwd unknown /proc/60968/task/61394/cwd (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61394 safe_time ceph rtd unknown /proc/60968/task/61394/root (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61394 safe_time ceph txt unknown /proc/60968/task/61394/exe (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61394 safe_time ceph NOFD /proc/60968/task/61394/fd (opendir: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61395 safe_time ceph cwd unknown /proc/60968/task/61395/cwd (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61395 safe_time ceph rtd unknown /proc/60968/task/61395/root (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61395 safe_time ceph txt unknown /proc/60968/task/61395/exe (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61395 safe_time ceph NOFD /proc/60968/task/61395/fd (opendir: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61396 safe_time ceph cwd unknown /proc/60968/task/61396/cwd (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61396 safe_time ceph rtd unknown /proc/60968/task/61396/root (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61396 safe_time ceph txt unknown /proc/60968/task/61396/exe (readlink: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61396 safe_time ceph NOFD /proc/60968/task/61396/fd (opendir: Permission denied) 2024-03-16T11:08:29.727 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61397 fn_anonym ceph cwd unknown /proc/60968/task/61397/cwd (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61397 fn_anonym ceph rtd unknown /proc/60968/task/61397/root (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61397 fn_anonym ceph txt unknown /proc/60968/task/61397/exe (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61397 fn_anonym ceph NOFD /proc/60968/task/61397/fd (opendir: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61403 rocksdb:l ceph cwd unknown /proc/60968/task/61403/cwd (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61403 rocksdb:l ceph rtd unknown /proc/60968/task/61403/root (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61403 rocksdb:l ceph txt unknown /proc/60968/task/61403/exe (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61403 rocksdb:l ceph NOFD /proc/60968/task/61403/fd (opendir: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61404 rocksdb:l ceph cwd unknown /proc/60968/task/61404/cwd (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61404 rocksdb:l ceph rtd unknown /proc/60968/task/61404/root (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61404 rocksdb:l ceph txt unknown /proc/60968/task/61404/exe (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61404 rocksdb:l ceph NOFD /proc/60968/task/61404/fd (opendir: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61405 rocksdb:l ceph cwd unknown /proc/60968/task/61405/cwd (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61405 rocksdb:l ceph rtd unknown /proc/60968/task/61405/root (readlink: Permission denied) 2024-03-16T11:08:29.728 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61405 rocksdb:l ceph txt unknown /proc/60968/task/61405/exe (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61405 rocksdb:l ceph NOFD /proc/60968/task/61405/fd (opendir: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61406 rocksdb:h ceph cwd unknown /proc/60968/task/61406/cwd (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61406 rocksdb:h ceph rtd unknown /proc/60968/task/61406/root (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61406 rocksdb:h ceph txt unknown /proc/60968/task/61406/exe (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61406 rocksdb:h ceph NOFD /proc/60968/task/61406/fd (opendir: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61885 bstore_ai ceph cwd unknown /proc/60968/task/61885/cwd (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61885 bstore_ai ceph rtd unknown /proc/60968/task/61885/root (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61885 bstore_ai ceph txt unknown /proc/60968/task/61885/exe (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 61885 bstore_ai ceph NOFD /proc/60968/task/61885/fd (opendir: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62070 bstore_ai ceph cwd unknown /proc/60968/task/62070/cwd (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62070 bstore_ai ceph rtd unknown /proc/60968/task/62070/root (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62070 bstore_ai ceph txt unknown /proc/60968/task/62070/exe (readlink: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62070 bstore_ai ceph NOFD /proc/60968/task/62070/fd (opendir: Permission denied) 2024-03-16T11:08:29.729 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62252 ceph-osd ceph cwd unknown /proc/60968/task/62252/cwd (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62252 ceph-osd ceph rtd unknown /proc/60968/task/62252/root (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62252 ceph-osd ceph txt unknown /proc/60968/task/62252/exe (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62252 ceph-osd ceph NOFD /proc/60968/task/62252/fd (opendir: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62253 cfin ceph cwd unknown /proc/60968/task/62253/cwd (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62253 cfin ceph rtd unknown /proc/60968/task/62253/root (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62253 cfin ceph txt unknown /proc/60968/task/62253/exe (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62253 cfin ceph NOFD /proc/60968/task/62253/fd (opendir: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62254 bstore_kv ceph cwd unknown /proc/60968/task/62254/cwd (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62254 bstore_kv ceph rtd unknown /proc/60968/task/62254/root (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62254 bstore_kv ceph txt unknown /proc/60968/task/62254/exe (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62254 bstore_kv ceph NOFD /proc/60968/task/62254/fd (opendir: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62255 bstore_kv ceph cwd unknown /proc/60968/task/62255/cwd (readlink: Permission denied) 2024-03-16T11:08:29.730 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62255 bstore_kv ceph rtd unknown /proc/60968/task/62255/root (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62255 bstore_kv ceph txt unknown /proc/60968/task/62255/exe (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62255 bstore_kv ceph NOFD /proc/60968/task/62255/fd (opendir: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62256 bstore_me ceph cwd unknown /proc/60968/task/62256/cwd (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62256 bstore_me ceph rtd unknown /proc/60968/task/62256/root (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62256 bstore_me ceph txt unknown /proc/60968/task/62256/exe (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62256 bstore_me ceph NOFD /proc/60968/task/62256/fd (opendir: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62257 ms_dispat ceph cwd unknown /proc/60968/task/62257/cwd (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62257 ms_dispat ceph rtd unknown /proc/60968/task/62257/root (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62257 ms_dispat ceph txt unknown /proc/60968/task/62257/exe (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62257 ms_dispat ceph NOFD /proc/60968/task/62257/fd (opendir: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62258 ms_local ceph cwd unknown /proc/60968/task/62258/cwd (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62258 ms_local ceph rtd unknown /proc/60968/task/62258/root (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62258 ms_local ceph txt unknown /proc/60968/task/62258/exe (readlink: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62258 ms_local ceph NOFD /proc/60968/task/62258/fd (opendir: Permission denied) 2024-03-16T11:08:29.731 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62259 safe_time ceph cwd unknown /proc/60968/task/62259/cwd (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62259 safe_time ceph rtd unknown /proc/60968/task/62259/root (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62259 safe_time ceph txt unknown /proc/60968/task/62259/exe (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62259 safe_time ceph NOFD /proc/60968/task/62259/fd (opendir: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62260 safe_time ceph cwd unknown /proc/60968/task/62260/cwd (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62260 safe_time ceph rtd unknown /proc/60968/task/62260/root (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62260 safe_time ceph txt unknown /proc/60968/task/62260/exe (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62260 safe_time ceph NOFD /proc/60968/task/62260/fd (opendir: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62261 ms_dispat ceph cwd unknown /proc/60968/task/62261/cwd (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62261 ms_dispat ceph rtd unknown /proc/60968/task/62261/root (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62261 ms_dispat ceph txt unknown /proc/60968/task/62261/exe (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62261 ms_dispat ceph NOFD /proc/60968/task/62261/fd (opendir: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62262 ms_local ceph cwd unknown /proc/60968/task/62262/cwd (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62262 ms_local ceph rtd unknown /proc/60968/task/62262/root (readlink: Permission denied) 2024-03-16T11:08:29.732 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62262 ms_local ceph txt unknown /proc/60968/task/62262/exe (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62262 ms_local ceph NOFD /proc/60968/task/62262/fd (opendir: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62263 ms_dispat ceph cwd unknown /proc/60968/task/62263/cwd (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62263 ms_dispat ceph rtd unknown /proc/60968/task/62263/root (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62263 ms_dispat ceph txt unknown /proc/60968/task/62263/exe (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62263 ms_dispat ceph NOFD /proc/60968/task/62263/fd (opendir: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62264 ms_local ceph cwd unknown /proc/60968/task/62264/cwd (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62264 ms_local ceph rtd unknown /proc/60968/task/62264/root (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62264 ms_local ceph txt unknown /proc/60968/task/62264/exe (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62264 ms_local ceph NOFD /proc/60968/task/62264/fd (opendir: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62265 ms_dispat ceph cwd unknown /proc/60968/task/62265/cwd (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62265 ms_dispat ceph rtd unknown /proc/60968/task/62265/root (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62265 ms_dispat ceph txt unknown /proc/60968/task/62265/exe (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62265 ms_dispat ceph NOFD /proc/60968/task/62265/fd (opendir: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62266 ms_local ceph cwd unknown /proc/60968/task/62266/cwd (readlink: Permission denied) 2024-03-16T11:08:29.733 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62266 ms_local ceph rtd unknown /proc/60968/task/62266/root (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62266 ms_local ceph txt unknown /proc/60968/task/62266/exe (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62266 ms_local ceph NOFD /proc/60968/task/62266/fd (opendir: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62267 ms_dispat ceph cwd unknown /proc/60968/task/62267/cwd (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62267 ms_dispat ceph rtd unknown /proc/60968/task/62267/root (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62267 ms_dispat ceph txt unknown /proc/60968/task/62267/exe (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62267 ms_dispat ceph NOFD /proc/60968/task/62267/fd (opendir: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62268 ms_local ceph cwd unknown /proc/60968/task/62268/cwd (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62268 ms_local ceph rtd unknown /proc/60968/task/62268/root (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62268 ms_local ceph txt unknown /proc/60968/task/62268/exe (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62268 ms_local ceph NOFD /proc/60968/task/62268/fd (opendir: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62269 ms_dispat ceph cwd unknown /proc/60968/task/62269/cwd (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62269 ms_dispat ceph rtd unknown /proc/60968/task/62269/root (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62269 ms_dispat ceph txt unknown /proc/60968/task/62269/exe (readlink: Permission denied) 2024-03-16T11:08:29.734 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62269 ms_dispat ceph NOFD /proc/60968/task/62269/fd (opendir: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62270 ms_local ceph cwd unknown /proc/60968/task/62270/cwd (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62270 ms_local ceph rtd unknown /proc/60968/task/62270/root (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62270 ms_local ceph txt unknown /proc/60968/task/62270/exe (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62270 ms_local ceph NOFD /proc/60968/task/62270/fd (opendir: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62271 ms_dispat ceph cwd unknown /proc/60968/task/62271/cwd (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62271 ms_dispat ceph rtd unknown /proc/60968/task/62271/root (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62271 ms_dispat ceph txt unknown /proc/60968/task/62271/exe (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62271 ms_dispat ceph NOFD /proc/60968/task/62271/fd (opendir: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62272 ms_local ceph cwd unknown /proc/60968/task/62272/cwd (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62272 ms_local ceph rtd unknown /proc/60968/task/62272/root (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62272 ms_local ceph txt unknown /proc/60968/task/62272/exe (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62272 ms_local ceph NOFD /proc/60968/task/62272/fd (opendir: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62273 fn_anonym ceph cwd unknown /proc/60968/task/62273/cwd (readlink: Permission denied) 2024-03-16T11:08:29.735 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62273 fn_anonym ceph rtd unknown /proc/60968/task/62273/root (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62273 fn_anonym ceph txt unknown /proc/60968/task/62273/exe (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62273 fn_anonym ceph NOFD /proc/60968/task/62273/fd (opendir: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62274 finisher ceph cwd unknown /proc/60968/task/62274/cwd (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62274 finisher ceph rtd unknown /proc/60968/task/62274/root (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62274 finisher ceph txt unknown /proc/60968/task/62274/exe (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62274 finisher ceph NOFD /proc/60968/task/62274/fd (opendir: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62275 safe_time ceph cwd unknown /proc/60968/task/62275/cwd (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62275 safe_time ceph rtd unknown /proc/60968/task/62275/root (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62275 safe_time ceph txt unknown /proc/60968/task/62275/exe (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62275 safe_time ceph NOFD /proc/60968/task/62275/fd (opendir: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62276 safe_time ceph cwd unknown /proc/60968/task/62276/cwd (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62276 safe_time ceph rtd unknown /proc/60968/task/62276/root (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62276 safe_time ceph txt unknown /proc/60968/task/62276/exe (readlink: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62276 safe_time ceph NOFD /proc/60968/task/62276/fd (opendir: Permission denied) 2024-03-16T11:08:29.736 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62277 ceph-osd ceph cwd unknown /proc/60968/task/62277/cwd (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62277 ceph-osd ceph rtd unknown /proc/60968/task/62277/root (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62277 ceph-osd ceph txt unknown /proc/60968/task/62277/exe (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62277 ceph-osd ceph NOFD /proc/60968/task/62277/fd (opendir: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62278 osd_srv_a ceph cwd unknown /proc/60968/task/62278/cwd (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62278 osd_srv_a ceph rtd unknown /proc/60968/task/62278/root (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62278 osd_srv_a ceph txt unknown /proc/60968/task/62278/exe (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62278 osd_srv_a ceph NOFD /proc/60968/task/62278/fd (opendir: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62279 tp_osd_tp ceph cwd unknown /proc/60968/task/62279/cwd (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62279 tp_osd_tp ceph rtd unknown /proc/60968/task/62279/root (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62279 tp_osd_tp ceph txt unknown /proc/60968/task/62279/exe (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62279 tp_osd_tp ceph NOFD /proc/60968/task/62279/fd (opendir: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62280 tp_osd_tp ceph cwd unknown /proc/60968/task/62280/cwd (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62280 tp_osd_tp ceph rtd unknown /proc/60968/task/62280/root (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62280 tp_osd_tp ceph txt unknown /proc/60968/task/62280/exe (readlink: Permission denied) 2024-03-16T11:08:29.737 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62280 tp_osd_tp ceph NOFD /proc/60968/task/62280/fd (opendir: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62281 tp_osd_tp ceph cwd unknown /proc/60968/task/62281/cwd (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62281 tp_osd_tp ceph rtd unknown /proc/60968/task/62281/root (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62281 tp_osd_tp ceph txt unknown /proc/60968/task/62281/exe (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62281 tp_osd_tp ceph NOFD /proc/60968/task/62281/fd (opendir: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62282 tp_osd_tp ceph cwd unknown /proc/60968/task/62282/cwd (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62282 tp_osd_tp ceph rtd unknown /proc/60968/task/62282/root (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62282 tp_osd_tp ceph txt unknown /proc/60968/task/62282/exe (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62282 tp_osd_tp ceph NOFD /proc/60968/task/62282/fd (opendir: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62283 tp_osd_tp ceph cwd unknown /proc/60968/task/62283/cwd (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62283 tp_osd_tp ceph rtd unknown /proc/60968/task/62283/root (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62283 tp_osd_tp ceph txt unknown /proc/60968/task/62283/exe (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62283 tp_osd_tp ceph NOFD /proc/60968/task/62283/fd (opendir: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62284 tp_osd_tp ceph cwd unknown /proc/60968/task/62284/cwd (readlink: Permission denied) 2024-03-16T11:08:29.738 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62284 tp_osd_tp ceph rtd unknown /proc/60968/task/62284/root (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62284 tp_osd_tp ceph txt unknown /proc/60968/task/62284/exe (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62284 tp_osd_tp ceph NOFD /proc/60968/task/62284/fd (opendir: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62285 tp_osd_tp ceph cwd unknown /proc/60968/task/62285/cwd (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62285 tp_osd_tp ceph rtd unknown /proc/60968/task/62285/root (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62285 tp_osd_tp ceph txt unknown /proc/60968/task/62285/exe (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62285 tp_osd_tp ceph NOFD /proc/60968/task/62285/fd (opendir: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62286 tp_osd_tp ceph cwd unknown /proc/60968/task/62286/cwd (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62286 tp_osd_tp ceph rtd unknown /proc/60968/task/62286/root (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62286 tp_osd_tp ceph txt unknown /proc/60968/task/62286/exe (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62286 tp_osd_tp ceph NOFD /proc/60968/task/62286/fd (opendir: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62287 tp_osd_tp ceph cwd unknown /proc/60968/task/62287/cwd (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62287 tp_osd_tp ceph rtd unknown /proc/60968/task/62287/root (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62287 tp_osd_tp ceph txt unknown /proc/60968/task/62287/exe (readlink: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62287 tp_osd_tp ceph NOFD /proc/60968/task/62287/fd (opendir: Permission denied) 2024-03-16T11:08:29.739 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62288 tp_osd_tp ceph cwd unknown /proc/60968/task/62288/cwd (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62288 tp_osd_tp ceph rtd unknown /proc/60968/task/62288/root (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62288 tp_osd_tp ceph txt unknown /proc/60968/task/62288/exe (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62288 tp_osd_tp ceph NOFD /proc/60968/task/62288/fd (opendir: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62289 tp_osd_tp ceph cwd unknown /proc/60968/task/62289/cwd (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62289 tp_osd_tp ceph rtd unknown /proc/60968/task/62289/root (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62289 tp_osd_tp ceph txt unknown /proc/60968/task/62289/exe (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62289 tp_osd_tp ceph NOFD /proc/60968/task/62289/fd (opendir: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62290 tp_osd_tp ceph cwd unknown /proc/60968/task/62290/cwd (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62290 tp_osd_tp ceph rtd unknown /proc/60968/task/62290/root (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62290 tp_osd_tp ceph txt unknown /proc/60968/task/62290/exe (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62290 tp_osd_tp ceph NOFD /proc/60968/task/62290/fd (opendir: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62291 tp_osd_tp ceph cwd unknown /proc/60968/task/62291/cwd (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62291 tp_osd_tp ceph rtd unknown /proc/60968/task/62291/root (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62291 tp_osd_tp ceph txt unknown /proc/60968/task/62291/exe (readlink: Permission denied) 2024-03-16T11:08:29.740 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62291 tp_osd_tp ceph NOFD /proc/60968/task/62291/fd (opendir: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62292 tp_osd_tp ceph cwd unknown /proc/60968/task/62292/cwd (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62292 tp_osd_tp ceph rtd unknown /proc/60968/task/62292/root (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62292 tp_osd_tp ceph txt unknown /proc/60968/task/62292/exe (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62292 tp_osd_tp ceph NOFD /proc/60968/task/62292/fd (opendir: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62293 tp_osd_tp ceph cwd unknown /proc/60968/task/62293/cwd (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62293 tp_osd_tp ceph rtd unknown /proc/60968/task/62293/root (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62293 tp_osd_tp ceph txt unknown /proc/60968/task/62293/exe (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62293 tp_osd_tp ceph NOFD /proc/60968/task/62293/fd (opendir: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62294 tp_osd_tp ceph cwd unknown /proc/60968/task/62294/cwd (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62294 tp_osd_tp ceph rtd unknown /proc/60968/task/62294/root (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62294 tp_osd_tp ceph txt unknown /proc/60968/task/62294/exe (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62294 tp_osd_tp ceph NOFD /proc/60968/task/62294/fd (opendir: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62295 osd_srv_h ceph cwd unknown /proc/60968/task/62295/cwd (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62295 osd_srv_h ceph rtd unknown /proc/60968/task/62295/root (readlink: Permission denied) 2024-03-16T11:08:29.741 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62295 osd_srv_h ceph txt unknown /proc/60968/task/62295/exe (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 60968 62295 osd_srv_h ceph NOFD /proc/60968/task/62295/fd (opendir: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 62390 root cwd unknown /proc/62390/cwd (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 62390 root rtd unknown /proc/62390/root (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 62390 root txt unknown /proc/62390/exe (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 62390 root NOFD /proc/62390/fd (opendir: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 62416 root cwd unknown /proc/62416/cwd (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 62416 root rtd unknown /proc/62416/root (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 62416 root txt unknown /proc/62416/exe (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 62416 root NOFD /proc/62416/fd (opendir: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:anacron 62465 root cwd unknown /proc/62465/cwd (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:anacron 62465 root rtd unknown /proc/62465/root (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:anacron 62465 root txt unknown /proc/62465/exe (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:anacron 62465 root NOFD /proc/62465/fd (opendir: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 64728 root cwd unknown /proc/64728/cwd (readlink: Permission denied) 2024-03-16T11:08:29.742 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 64728 root rtd unknown /proc/64728/root (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 64728 root txt unknown /proc/64728/exe (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 64728 root NOFD /proc/64728/fd (opendir: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 67597 root cwd unknown /proc/67597/cwd (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 67597 root rtd unknown /proc/67597/root (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 67597 root txt unknown /proc/67597/exe (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 67597 root NOFD /proc/67597/fd (opendir: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 67599 root cwd unknown /proc/67599/cwd (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 67599 root rtd unknown /proc/67599/root (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 67599 root txt unknown /proc/67599/exe (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 67599 root NOFD /proc/67599/fd (opendir: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 ceph cwd unknown /proc/67601/cwd (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 ceph rtd unknown /proc/67601/root (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 ceph txt unknown /proc/67601/exe (readlink: Permission denied) 2024-03-16T11:08:29.743 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 ceph NOFD /proc/67601/fd (opendir: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67602 log ceph cwd unknown /proc/67601/task/67602/cwd (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67602 log ceph rtd unknown /proc/67601/task/67602/root (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67602 log ceph txt unknown /proc/67601/task/67602/exe (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67602 log ceph NOFD /proc/67601/task/67602/fd (opendir: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67604 msgr-work ceph cwd unknown /proc/67601/task/67604/cwd (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67604 msgr-work ceph rtd unknown /proc/67601/task/67604/root (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67604 msgr-work ceph txt unknown /proc/67601/task/67604/exe (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67604 msgr-work ceph NOFD /proc/67601/task/67604/fd (opendir: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67605 msgr-work ceph cwd unknown /proc/67601/task/67605/cwd (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67605 msgr-work ceph rtd unknown /proc/67601/task/67605/root (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67605 msgr-work ceph txt unknown /proc/67601/task/67605/exe (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67605 msgr-work ceph NOFD /proc/67601/task/67605/fd (opendir: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67606 msgr-work ceph cwd unknown /proc/67601/task/67606/cwd (readlink: Permission denied) 2024-03-16T11:08:29.744 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67606 msgr-work ceph rtd unknown /proc/67601/task/67606/root (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67606 msgr-work ceph txt unknown /proc/67601/task/67606/exe (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67606 msgr-work ceph NOFD /proc/67601/task/67606/fd (opendir: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67610 service ceph cwd unknown /proc/67601/task/67610/cwd (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67610 service ceph rtd unknown /proc/67601/task/67610/root (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67610 service ceph txt unknown /proc/67601/task/67610/exe (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67610 service ceph NOFD /proc/67601/task/67610/fd (opendir: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67611 admin_soc ceph cwd unknown /proc/67601/task/67611/cwd (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67611 admin_soc ceph rtd unknown /proc/67601/task/67611/root (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67611 admin_soc ceph txt unknown /proc/67601/task/67611/exe (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67611 admin_soc ceph NOFD /proc/67601/task/67611/fd (opendir: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67753 signal_ha ceph cwd unknown /proc/67601/task/67753/cwd (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67753 signal_ha ceph rtd unknown /proc/67601/task/67753/root (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67753 signal_ha ceph txt unknown /proc/67601/task/67753/exe (readlink: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67753 signal_ha ceph NOFD /proc/67601/task/67753/fd (opendir: Permission denied) 2024-03-16T11:08:29.745 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67754 io_contex ceph cwd unknown /proc/67601/task/67754/cwd (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67754 io_contex ceph rtd unknown /proc/67601/task/67754/root (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67754 io_contex ceph txt unknown /proc/67601/task/67754/exe (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67754 io_contex ceph NOFD /proc/67601/task/67754/fd (opendir: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67755 io_contex ceph cwd unknown /proc/67601/task/67755/cwd (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67755 io_contex ceph rtd unknown /proc/67601/task/67755/root (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67755 io_contex ceph txt unknown /proc/67601/task/67755/exe (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67755 io_contex ceph NOFD /proc/67601/task/67755/fd (opendir: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67867 OpHistory ceph cwd unknown /proc/67601/task/67867/cwd (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67867 OpHistory ceph rtd unknown /proc/67601/task/67867/root (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67867 OpHistory ceph txt unknown /proc/67601/task/67867/exe (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67867 OpHistory ceph NOFD /proc/67601/task/67867/fd (opendir: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67868 ceph_time ceph cwd unknown /proc/67601/task/67868/cwd (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67868 ceph_time ceph rtd unknown /proc/67601/task/67868/root (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67868 ceph_time ceph txt unknown /proc/67601/task/67868/exe (readlink: Permission denied) 2024-03-16T11:08:29.746 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67868 ceph_time ceph NOFD /proc/67601/task/67868/fd (opendir: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67916 ceph-osd ceph cwd unknown /proc/67601/task/67916/cwd (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67916 ceph-osd ceph rtd unknown /proc/67601/task/67916/root (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67916 ceph-osd ceph txt unknown /proc/67601/task/67916/exe (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67916 ceph-osd ceph NOFD /proc/67601/task/67916/fd (opendir: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67919 ceph-osd ceph cwd unknown /proc/67601/task/67919/cwd (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67919 ceph-osd ceph rtd unknown /proc/67601/task/67919/root (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67919 ceph-osd ceph txt unknown /proc/67601/task/67919/exe (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67919 ceph-osd ceph NOFD /proc/67601/task/67919/fd (opendir: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67925 ceph-osd ceph cwd unknown /proc/67601/task/67925/cwd (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67925 ceph-osd ceph rtd unknown /proc/67601/task/67925/root (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67925 ceph-osd ceph txt unknown /proc/67601/task/67925/exe (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67925 ceph-osd ceph NOFD /proc/67601/task/67925/fd (opendir: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67950 ceph-osd ceph cwd unknown /proc/67601/task/67950/cwd (readlink: Permission denied) 2024-03-16T11:08:29.747 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67950 ceph-osd ceph rtd unknown /proc/67601/task/67950/root (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67950 ceph-osd ceph txt unknown /proc/67601/task/67950/exe (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67950 ceph-osd ceph NOFD /proc/67601/task/67950/fd (opendir: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67970 ceph-osd ceph cwd unknown /proc/67601/task/67970/cwd (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67970 ceph-osd ceph rtd unknown /proc/67601/task/67970/root (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67970 ceph-osd ceph txt unknown /proc/67601/task/67970/exe (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67970 ceph-osd ceph NOFD /proc/67601/task/67970/fd (opendir: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67992 ceph-osd ceph cwd unknown /proc/67601/task/67992/cwd (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67992 ceph-osd ceph rtd unknown /proc/67601/task/67992/root (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67992 ceph-osd ceph txt unknown /proc/67601/task/67992/exe (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 67992 ceph-osd ceph NOFD /proc/67601/task/67992/fd (opendir: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68031 ceph-osd ceph cwd unknown /proc/67601/task/68031/cwd (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68031 ceph-osd ceph rtd unknown /proc/67601/task/68031/root (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68031 ceph-osd ceph txt unknown /proc/67601/task/68031/exe (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68031 ceph-osd ceph NOFD /proc/67601/task/68031/fd (opendir: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68083 ceph-osd ceph cwd unknown /proc/67601/task/68083/cwd (readlink: Permission denied) 2024-03-16T11:08:29.748 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68083 ceph-osd ceph rtd unknown /proc/67601/task/68083/root (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68083 ceph-osd ceph txt unknown /proc/67601/task/68083/exe (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68083 ceph-osd ceph NOFD /proc/67601/task/68083/fd (opendir: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68085 ceph-osd ceph cwd unknown /proc/67601/task/68085/cwd (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68085 ceph-osd ceph rtd unknown /proc/67601/task/68085/root (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68085 ceph-osd ceph txt unknown /proc/67601/task/68085/exe (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68085 ceph-osd ceph NOFD /proc/67601/task/68085/fd (opendir: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68086 safe_time ceph cwd unknown /proc/67601/task/68086/cwd (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68086 safe_time ceph rtd unknown /proc/67601/task/68086/root (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68086 safe_time ceph txt unknown /proc/67601/task/68086/exe (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68086 safe_time ceph NOFD /proc/67601/task/68086/fd (opendir: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68087 safe_time ceph cwd unknown /proc/67601/task/68087/cwd (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68087 safe_time ceph rtd unknown /proc/67601/task/68087/root (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68087 safe_time ceph txt unknown /proc/67601/task/68087/exe (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68087 safe_time ceph NOFD /proc/67601/task/68087/fd (opendir: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68088 safe_time ceph cwd unknown /proc/67601/task/68088/cwd (readlink: Permission denied) 2024-03-16T11:08:29.749 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68088 safe_time ceph rtd unknown /proc/67601/task/68088/root (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68088 safe_time ceph txt unknown /proc/67601/task/68088/exe (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68088 safe_time ceph NOFD /proc/67601/task/68088/fd (opendir: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68089 safe_time ceph cwd unknown /proc/67601/task/68089/cwd (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68089 safe_time ceph rtd unknown /proc/67601/task/68089/root (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68089 safe_time ceph txt unknown /proc/67601/task/68089/exe (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68089 safe_time ceph NOFD /proc/67601/task/68089/fd (opendir: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68090 fn_anonym ceph cwd unknown /proc/67601/task/68090/cwd (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68090 fn_anonym ceph rtd unknown /proc/67601/task/68090/root (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68090 fn_anonym ceph txt unknown /proc/67601/task/68090/exe (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68090 fn_anonym ceph NOFD /proc/67601/task/68090/fd (opendir: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68094 rocksdb:l ceph cwd unknown /proc/67601/task/68094/cwd (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68094 rocksdb:l ceph rtd unknown /proc/67601/task/68094/root (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68094 rocksdb:l ceph txt unknown /proc/67601/task/68094/exe (readlink: Permission denied) 2024-03-16T11:08:29.750 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68094 rocksdb:l ceph NOFD /proc/67601/task/68094/fd (opendir: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68095 rocksdb:l ceph cwd unknown /proc/67601/task/68095/cwd (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68095 rocksdb:l ceph rtd unknown /proc/67601/task/68095/root (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68095 rocksdb:l ceph txt unknown /proc/67601/task/68095/exe (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68095 rocksdb:l ceph NOFD /proc/67601/task/68095/fd (opendir: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68096 rocksdb:l ceph cwd unknown /proc/67601/task/68096/cwd (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68096 rocksdb:l ceph rtd unknown /proc/67601/task/68096/root (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68096 rocksdb:l ceph txt unknown /proc/67601/task/68096/exe (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68096 rocksdb:l ceph NOFD /proc/67601/task/68096/fd (opendir: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68097 rocksdb:h ceph cwd unknown /proc/67601/task/68097/cwd (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68097 rocksdb:h ceph rtd unknown /proc/67601/task/68097/root (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68097 rocksdb:h ceph txt unknown /proc/67601/task/68097/exe (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68097 rocksdb:h ceph NOFD /proc/67601/task/68097/fd (opendir: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68534 bstore_ai ceph cwd unknown /proc/67601/task/68534/cwd (readlink: Permission denied) 2024-03-16T11:08:29.751 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68534 bstore_ai ceph rtd unknown /proc/67601/task/68534/root (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68534 bstore_ai ceph txt unknown /proc/67601/task/68534/exe (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68534 bstore_ai ceph NOFD /proc/67601/task/68534/fd (opendir: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68719 bstore_ai ceph cwd unknown /proc/67601/task/68719/cwd (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68719 bstore_ai ceph rtd unknown /proc/67601/task/68719/root (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68719 bstore_ai ceph txt unknown /proc/67601/task/68719/exe (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68719 bstore_ai ceph NOFD /proc/67601/task/68719/fd (opendir: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68917 ceph-osd ceph cwd unknown /proc/67601/task/68917/cwd (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68917 ceph-osd ceph rtd unknown /proc/67601/task/68917/root (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68917 ceph-osd ceph txt unknown /proc/67601/task/68917/exe (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68917 ceph-osd ceph NOFD /proc/67601/task/68917/fd (opendir: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68920 cfin ceph cwd unknown /proc/67601/task/68920/cwd (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68920 cfin ceph rtd unknown /proc/67601/task/68920/root (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68920 cfin ceph txt unknown /proc/67601/task/68920/exe (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68920 cfin ceph NOFD /proc/67601/task/68920/fd (opendir: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68921 bstore_kv ceph cwd unknown /proc/67601/task/68921/cwd (readlink: Permission denied) 2024-03-16T11:08:29.752 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68921 bstore_kv ceph rtd unknown /proc/67601/task/68921/root (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68921 bstore_kv ceph txt unknown /proc/67601/task/68921/exe (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68921 bstore_kv ceph NOFD /proc/67601/task/68921/fd (opendir: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68922 bstore_kv ceph cwd unknown /proc/67601/task/68922/cwd (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68922 bstore_kv ceph rtd unknown /proc/67601/task/68922/root (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68922 bstore_kv ceph txt unknown /proc/67601/task/68922/exe (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68922 bstore_kv ceph NOFD /proc/67601/task/68922/fd (opendir: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68923 bstore_me ceph cwd unknown /proc/67601/task/68923/cwd (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68923 bstore_me ceph rtd unknown /proc/67601/task/68923/root (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68923 bstore_me ceph txt unknown /proc/67601/task/68923/exe (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68923 bstore_me ceph NOFD /proc/67601/task/68923/fd (opendir: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68936 ms_dispat ceph cwd unknown /proc/67601/task/68936/cwd (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68936 ms_dispat ceph rtd unknown /proc/67601/task/68936/root (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68936 ms_dispat ceph txt unknown /proc/67601/task/68936/exe (readlink: Permission denied) 2024-03-16T11:08:29.753 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68936 ms_dispat ceph NOFD /proc/67601/task/68936/fd (opendir: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68937 ms_local ceph cwd unknown /proc/67601/task/68937/cwd (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68937 ms_local ceph rtd unknown /proc/67601/task/68937/root (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68937 ms_local ceph txt unknown /proc/67601/task/68937/exe (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68937 ms_local ceph NOFD /proc/67601/task/68937/fd (opendir: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68938 safe_time ceph cwd unknown /proc/67601/task/68938/cwd (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68938 safe_time ceph rtd unknown /proc/67601/task/68938/root (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68938 safe_time ceph txt unknown /proc/67601/task/68938/exe (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68938 safe_time ceph NOFD /proc/67601/task/68938/fd (opendir: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68939 safe_time ceph cwd unknown /proc/67601/task/68939/cwd (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68939 safe_time ceph rtd unknown /proc/67601/task/68939/root (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68939 safe_time ceph txt unknown /proc/67601/task/68939/exe (readlink: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68939 safe_time ceph NOFD /proc/67601/task/68939/fd (opendir: Permission denied) 2024-03-16T11:08:29.754 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68940 ms_dispat ceph cwd unknown /proc/67601/task/68940/cwd (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68940 ms_dispat ceph rtd unknown /proc/67601/task/68940/root (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68940 ms_dispat ceph txt unknown /proc/67601/task/68940/exe (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68940 ms_dispat ceph NOFD /proc/67601/task/68940/fd (opendir: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68941 ms_local ceph cwd unknown /proc/67601/task/68941/cwd (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68941 ms_local ceph rtd unknown /proc/67601/task/68941/root (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68941 ms_local ceph txt unknown /proc/67601/task/68941/exe (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68941 ms_local ceph NOFD /proc/67601/task/68941/fd (opendir: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68942 ms_dispat ceph cwd unknown /proc/67601/task/68942/cwd (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68942 ms_dispat ceph rtd unknown /proc/67601/task/68942/root (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68942 ms_dispat ceph txt unknown /proc/67601/task/68942/exe (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68942 ms_dispat ceph NOFD /proc/67601/task/68942/fd (opendir: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68944 ms_local ceph cwd unknown /proc/67601/task/68944/cwd (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68944 ms_local ceph rtd unknown /proc/67601/task/68944/root (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68944 ms_local ceph txt unknown /proc/67601/task/68944/exe (readlink: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68944 ms_local ceph NOFD /proc/67601/task/68944/fd (opendir: Permission denied) 2024-03-16T11:08:29.755 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68945 ms_dispat ceph cwd unknown /proc/67601/task/68945/cwd (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68945 ms_dispat ceph rtd unknown /proc/67601/task/68945/root (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68945 ms_dispat ceph txt unknown /proc/67601/task/68945/exe (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68945 ms_dispat ceph NOFD /proc/67601/task/68945/fd (opendir: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68946 ms_local ceph cwd unknown /proc/67601/task/68946/cwd (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68946 ms_local ceph rtd unknown /proc/67601/task/68946/root (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68946 ms_local ceph txt unknown /proc/67601/task/68946/exe (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68946 ms_local ceph NOFD /proc/67601/task/68946/fd (opendir: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68947 ms_dispat ceph cwd unknown /proc/67601/task/68947/cwd (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68947 ms_dispat ceph rtd unknown /proc/67601/task/68947/root (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68947 ms_dispat ceph txt unknown /proc/67601/task/68947/exe (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68947 ms_dispat ceph NOFD /proc/67601/task/68947/fd (opendir: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68948 ms_local ceph cwd unknown /proc/67601/task/68948/cwd (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68948 ms_local ceph rtd unknown /proc/67601/task/68948/root (readlink: Permission denied) 2024-03-16T11:08:29.756 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68948 ms_local ceph txt unknown /proc/67601/task/68948/exe (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68948 ms_local ceph NOFD /proc/67601/task/68948/fd (opendir: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68949 ms_dispat ceph cwd unknown /proc/67601/task/68949/cwd (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68949 ms_dispat ceph rtd unknown /proc/67601/task/68949/root (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68949 ms_dispat ceph txt unknown /proc/67601/task/68949/exe (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68949 ms_dispat ceph NOFD /proc/67601/task/68949/fd (opendir: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68950 ms_local ceph cwd unknown /proc/67601/task/68950/cwd (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68950 ms_local ceph rtd unknown /proc/67601/task/68950/root (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68950 ms_local ceph txt unknown /proc/67601/task/68950/exe (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68950 ms_local ceph NOFD /proc/67601/task/68950/fd (opendir: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68951 ms_dispat ceph cwd unknown /proc/67601/task/68951/cwd (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68951 ms_dispat ceph rtd unknown /proc/67601/task/68951/root (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68951 ms_dispat ceph txt unknown /proc/67601/task/68951/exe (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68951 ms_dispat ceph NOFD /proc/67601/task/68951/fd (opendir: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68952 ms_local ceph cwd unknown /proc/67601/task/68952/cwd (readlink: Permission denied) 2024-03-16T11:08:29.757 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68952 ms_local ceph rtd unknown /proc/67601/task/68952/root (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68952 ms_local ceph txt unknown /proc/67601/task/68952/exe (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68952 ms_local ceph NOFD /proc/67601/task/68952/fd (opendir: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68953 fn_anonym ceph cwd unknown /proc/67601/task/68953/cwd (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68953 fn_anonym ceph rtd unknown /proc/67601/task/68953/root (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68953 fn_anonym ceph txt unknown /proc/67601/task/68953/exe (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68953 fn_anonym ceph NOFD /proc/67601/task/68953/fd (opendir: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68954 finisher ceph cwd unknown /proc/67601/task/68954/cwd (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68954 finisher ceph rtd unknown /proc/67601/task/68954/root (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68954 finisher ceph txt unknown /proc/67601/task/68954/exe (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68954 finisher ceph NOFD /proc/67601/task/68954/fd (opendir: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68955 safe_time ceph cwd unknown /proc/67601/task/68955/cwd (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68955 safe_time ceph rtd unknown /proc/67601/task/68955/root (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68955 safe_time ceph txt unknown /proc/67601/task/68955/exe (readlink: Permission denied) 2024-03-16T11:08:29.758 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68955 safe_time ceph NOFD /proc/67601/task/68955/fd (opendir: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68956 safe_time ceph cwd unknown /proc/67601/task/68956/cwd (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68956 safe_time ceph rtd unknown /proc/67601/task/68956/root (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68956 safe_time ceph txt unknown /proc/67601/task/68956/exe (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68956 safe_time ceph NOFD /proc/67601/task/68956/fd (opendir: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68957 ceph-osd ceph cwd unknown /proc/67601/task/68957/cwd (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68957 ceph-osd ceph rtd unknown /proc/67601/task/68957/root (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68957 ceph-osd ceph txt unknown /proc/67601/task/68957/exe (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68957 ceph-osd ceph NOFD /proc/67601/task/68957/fd (opendir: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68958 osd_srv_a ceph cwd unknown /proc/67601/task/68958/cwd (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68958 osd_srv_a ceph rtd unknown /proc/67601/task/68958/root (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68958 osd_srv_a ceph txt unknown /proc/67601/task/68958/exe (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68958 osd_srv_a ceph NOFD /proc/67601/task/68958/fd (opendir: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68959 tp_osd_tp ceph cwd unknown /proc/67601/task/68959/cwd (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68959 tp_osd_tp ceph rtd unknown /proc/67601/task/68959/root (readlink: Permission denied) 2024-03-16T11:08:29.759 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68959 tp_osd_tp ceph txt unknown /proc/67601/task/68959/exe (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68959 tp_osd_tp ceph NOFD /proc/67601/task/68959/fd (opendir: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68960 tp_osd_tp ceph cwd unknown /proc/67601/task/68960/cwd (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68960 tp_osd_tp ceph rtd unknown /proc/67601/task/68960/root (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68960 tp_osd_tp ceph txt unknown /proc/67601/task/68960/exe (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68960 tp_osd_tp ceph NOFD /proc/67601/task/68960/fd (opendir: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68961 tp_osd_tp ceph cwd unknown /proc/67601/task/68961/cwd (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68961 tp_osd_tp ceph rtd unknown /proc/67601/task/68961/root (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68961 tp_osd_tp ceph txt unknown /proc/67601/task/68961/exe (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68961 tp_osd_tp ceph NOFD /proc/67601/task/68961/fd (opendir: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68962 tp_osd_tp ceph cwd unknown /proc/67601/task/68962/cwd (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68962 tp_osd_tp ceph rtd unknown /proc/67601/task/68962/root (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68962 tp_osd_tp ceph txt unknown /proc/67601/task/68962/exe (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68962 tp_osd_tp ceph NOFD /proc/67601/task/68962/fd (opendir: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68963 tp_osd_tp ceph cwd unknown /proc/67601/task/68963/cwd (readlink: Permission denied) 2024-03-16T11:08:29.760 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68963 tp_osd_tp ceph rtd unknown /proc/67601/task/68963/root (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68963 tp_osd_tp ceph txt unknown /proc/67601/task/68963/exe (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68963 tp_osd_tp ceph NOFD /proc/67601/task/68963/fd (opendir: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68964 tp_osd_tp ceph cwd unknown /proc/67601/task/68964/cwd (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68964 tp_osd_tp ceph rtd unknown /proc/67601/task/68964/root (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68964 tp_osd_tp ceph txt unknown /proc/67601/task/68964/exe (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68964 tp_osd_tp ceph NOFD /proc/67601/task/68964/fd (opendir: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68965 tp_osd_tp ceph cwd unknown /proc/67601/task/68965/cwd (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68965 tp_osd_tp ceph rtd unknown /proc/67601/task/68965/root (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68965 tp_osd_tp ceph txt unknown /proc/67601/task/68965/exe (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68965 tp_osd_tp ceph NOFD /proc/67601/task/68965/fd (opendir: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68966 tp_osd_tp ceph cwd unknown /proc/67601/task/68966/cwd (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68966 tp_osd_tp ceph rtd unknown /proc/67601/task/68966/root (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68966 tp_osd_tp ceph txt unknown /proc/67601/task/68966/exe (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68966 tp_osd_tp ceph NOFD /proc/67601/task/68966/fd (opendir: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68967 tp_osd_tp ceph cwd unknown /proc/67601/task/68967/cwd (readlink: Permission denied) 2024-03-16T11:08:29.761 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68967 tp_osd_tp ceph rtd unknown /proc/67601/task/68967/root (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68967 tp_osd_tp ceph txt unknown /proc/67601/task/68967/exe (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68967 tp_osd_tp ceph NOFD /proc/67601/task/68967/fd (opendir: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68968 tp_osd_tp ceph cwd unknown /proc/67601/task/68968/cwd (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68968 tp_osd_tp ceph rtd unknown /proc/67601/task/68968/root (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68968 tp_osd_tp ceph txt unknown /proc/67601/task/68968/exe (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68968 tp_osd_tp ceph NOFD /proc/67601/task/68968/fd (opendir: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68969 tp_osd_tp ceph cwd unknown /proc/67601/task/68969/cwd (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68969 tp_osd_tp ceph rtd unknown /proc/67601/task/68969/root (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68969 tp_osd_tp ceph txt unknown /proc/67601/task/68969/exe (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68969 tp_osd_tp ceph NOFD /proc/67601/task/68969/fd (opendir: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68970 tp_osd_tp ceph cwd unknown /proc/67601/task/68970/cwd (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68970 tp_osd_tp ceph rtd unknown /proc/67601/task/68970/root (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68970 tp_osd_tp ceph txt unknown /proc/67601/task/68970/exe (readlink: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68970 tp_osd_tp ceph NOFD /proc/67601/task/68970/fd (opendir: Permission denied) 2024-03-16T11:08:29.762 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68971 tp_osd_tp ceph cwd unknown /proc/67601/task/68971/cwd (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68971 tp_osd_tp ceph rtd unknown /proc/67601/task/68971/root (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68971 tp_osd_tp ceph txt unknown /proc/67601/task/68971/exe (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68971 tp_osd_tp ceph NOFD /proc/67601/task/68971/fd (opendir: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68973 tp_osd_tp ceph cwd unknown /proc/67601/task/68973/cwd (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68973 tp_osd_tp ceph rtd unknown /proc/67601/task/68973/root (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68973 tp_osd_tp ceph txt unknown /proc/67601/task/68973/exe (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68973 tp_osd_tp ceph NOFD /proc/67601/task/68973/fd (opendir: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68974 tp_osd_tp ceph cwd unknown /proc/67601/task/68974/cwd (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68974 tp_osd_tp ceph rtd unknown /proc/67601/task/68974/root (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68974 tp_osd_tp ceph txt unknown /proc/67601/task/68974/exe (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68974 tp_osd_tp ceph NOFD /proc/67601/task/68974/fd (opendir: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68975 tp_osd_tp ceph cwd unknown /proc/67601/task/68975/cwd (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68975 tp_osd_tp ceph rtd unknown /proc/67601/task/68975/root (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68975 tp_osd_tp ceph txt unknown /proc/67601/task/68975/exe (readlink: Permission denied) 2024-03-16T11:08:29.763 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68975 tp_osd_tp ceph NOFD /proc/67601/task/68975/fd (opendir: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68976 osd_srv_h ceph cwd unknown /proc/67601/task/68976/cwd (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68976 osd_srv_h ceph rtd unknown /proc/67601/task/68976/root (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68976 osd_srv_h ceph txt unknown /proc/67601/task/68976/exe (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-osd 67601 68976 osd_srv_h ceph NOFD /proc/67601/task/68976/fd (opendir: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 69040 root cwd unknown /proc/69040/cwd (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 69040 root rtd unknown /proc/69040/root (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 69040 root txt unknown /proc/69040/exe (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 69040 root NOFD /proc/69040/fd (opendir: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 69057 root cwd unknown /proc/69057/cwd (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 69057 root rtd unknown /proc/69057/root (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 69057 root txt unknown /proc/69057/exe (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 69057 root NOFD /proc/69057/fd (opendir: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72683 root cwd unknown /proc/72683/cwd (readlink: Permission denied) 2024-03-16T11:08:29.764 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72683 root rtd unknown /proc/72683/root (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72683 root txt unknown /proc/72683/exe (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72683 root NOFD /proc/72683/fd (opendir: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72700 root cwd unknown /proc/72700/cwd (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72700 root rtd unknown /proc/72700/root (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72700 root txt unknown /proc/72700/exe (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72700 root NOFD /proc/72700/fd (opendir: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72701 root cwd unknown /proc/72701/cwd (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72701 root rtd unknown /proc/72701/root (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72701 root txt unknown /proc/72701/exe (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72701 root NOFD /proc/72701/fd (opendir: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72718 root cwd unknown /proc/72718/cwd (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72718 root rtd unknown /proc/72718/root (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72718 root txt unknown /proc/72718/exe (readlink: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72718 root NOFD /proc/72718/fd (opendir: Permission denied) 2024-03-16T11:08:29.765 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72719 root cwd unknown /proc/72719/cwd (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72719 root rtd unknown /proc/72719/root (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72719 root txt unknown /proc/72719/exe (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72719 root NOFD /proc/72719/fd (opendir: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72736 root cwd unknown /proc/72736/cwd (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72736 root rtd unknown /proc/72736/root (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72736 root txt unknown /proc/72736/exe (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72736 root NOFD /proc/72736/fd (opendir: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72737 root cwd unknown /proc/72737/cwd (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72737 root rtd unknown /proc/72737/root (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72737 root txt unknown /proc/72737/exe (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 72737 root NOFD /proc/72737/fd (opendir: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72754 root cwd unknown /proc/72754/cwd (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72754 root rtd unknown /proc/72754/root (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72754 root txt unknown /proc/72754/exe (readlink: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:journalct 72754 root NOFD /proc/72754/fd (opendir: Permission denied) 2024-03-16T11:08:29.766 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73246 root cwd unknown /proc/73246/cwd (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73246 root rtd unknown /proc/73246/root (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73246 root txt unknown /proc/73246/exe (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73246 root NOFD /proc/73246/fd (opendir: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73248 root cwd unknown /proc/73248/cwd (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73248 root rtd unknown /proc/73248/root (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73248 root txt unknown /proc/73248/exe (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73248 root NOFD /proc/73248/fd (opendir: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 ceph cwd unknown /proc/73250/cwd (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 ceph rtd unknown /proc/73250/root (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 ceph txt unknown /proc/73250/exe (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 ceph NOFD /proc/73250/fd (opendir: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73265 log ceph cwd unknown /proc/73250/task/73265/cwd (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73265 log ceph rtd unknown /proc/73250/task/73265/root (readlink: Permission denied) 2024-03-16T11:08:29.767 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73265 log ceph txt unknown /proc/73250/task/73265/exe (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73265 log ceph NOFD /proc/73250/task/73265/fd (opendir: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73269 msgr-work ceph cwd unknown /proc/73250/task/73269/cwd (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73269 msgr-work ceph rtd unknown /proc/73250/task/73269/root (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73269 msgr-work ceph txt unknown /proc/73250/task/73269/exe (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73269 msgr-work ceph NOFD /proc/73250/task/73269/fd (opendir: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73270 msgr-work ceph cwd unknown /proc/73250/task/73270/cwd (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73270 msgr-work ceph rtd unknown /proc/73250/task/73270/root (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73270 msgr-work ceph txt unknown /proc/73250/task/73270/exe (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73270 msgr-work ceph NOFD /proc/73250/task/73270/fd (opendir: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73271 msgr-work ceph cwd unknown /proc/73250/task/73271/cwd (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73271 msgr-work ceph rtd unknown /proc/73250/task/73271/root (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73271 msgr-work ceph txt unknown /proc/73250/task/73271/exe (readlink: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73271 msgr-work ceph NOFD /proc/73250/task/73271/fd (opendir: Permission denied) 2024-03-16T11:08:29.768 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73275 service ceph cwd unknown /proc/73250/task/73275/cwd (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73275 service ceph rtd unknown /proc/73250/task/73275/root (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73275 service ceph txt unknown /proc/73250/task/73275/exe (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73275 service ceph NOFD /proc/73250/task/73275/fd (opendir: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73276 admin_soc ceph cwd unknown /proc/73250/task/73276/cwd (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73276 admin_soc ceph rtd unknown /proc/73250/task/73276/root (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73276 admin_soc ceph txt unknown /proc/73250/task/73276/exe (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73276 admin_soc ceph NOFD /proc/73250/task/73276/fd (opendir: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73277 signal_ha ceph cwd unknown /proc/73250/task/73277/cwd (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73277 signal_ha ceph rtd unknown /proc/73250/task/73277/root (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73277 signal_ha ceph txt unknown /proc/73250/task/73277/exe (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73277 signal_ha ceph NOFD /proc/73250/task/73277/fd (opendir: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73278 io_contex ceph cwd unknown /proc/73250/task/73278/cwd (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73278 io_contex ceph rtd unknown /proc/73250/task/73278/root (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73278 io_contex ceph txt unknown /proc/73250/task/73278/exe (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73278 io_contex ceph NOFD /proc/73250/task/73278/fd (opendir: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73279 io_contex ceph cwd unknown /proc/73250/task/73279/cwd (readlink: Permission denied) 2024-03-16T11:08:29.769 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73279 io_contex ceph rtd unknown /proc/73250/task/73279/root (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73279 io_contex ceph txt unknown /proc/73250/task/73279/exe (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73279 io_contex ceph NOFD /proc/73250/task/73279/fd (opendir: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73280 ms_dispat ceph cwd unknown /proc/73250/task/73280/cwd (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73280 ms_dispat ceph rtd unknown /proc/73250/task/73280/root (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73280 ms_dispat ceph txt unknown /proc/73250/task/73280/exe (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73280 ms_dispat ceph NOFD /proc/73250/task/73280/fd (opendir: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73281 ms_local ceph cwd unknown /proc/73250/task/73281/cwd (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73281 ms_local ceph rtd unknown /proc/73250/task/73281/root (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73281 ms_local ceph txt unknown /proc/73250/task/73281/exe (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73281 ms_local ceph NOFD /proc/73250/task/73281/fd (opendir: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73282 safe_time ceph cwd unknown /proc/73250/task/73282/cwd (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73282 safe_time ceph rtd unknown /proc/73250/task/73282/root (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73282 safe_time ceph txt unknown /proc/73250/task/73282/exe (readlink: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73282 safe_time ceph NOFD /proc/73250/task/73282/fd (opendir: Permission denied) 2024-03-16T11:08:29.770 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73283 safe_time ceph cwd unknown /proc/73250/task/73283/cwd (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73283 safe_time ceph rtd unknown /proc/73250/task/73283/root (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73283 safe_time ceph txt unknown /proc/73250/task/73283/exe (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73283 safe_time ceph NOFD /proc/73250/task/73283/fd (opendir: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73284 ceph-mds ceph cwd unknown /proc/73250/task/73284/cwd (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73284 ceph-mds ceph rtd unknown /proc/73250/task/73284/root (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73284 ceph-mds ceph txt unknown /proc/73250/task/73284/exe (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73284 ceph-mds ceph NOFD /proc/73250/task/73284/fd (opendir: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73437 safe_time ceph cwd unknown /proc/73250/task/73437/cwd (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73437 safe_time ceph rtd unknown /proc/73250/task/73437/root (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73437 safe_time ceph txt unknown /proc/73250/task/73437/exe (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73250 73437 safe_time ceph NOFD /proc/73250/task/73437/fd (opendir: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73634 root cwd unknown /proc/73634/cwd (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73634 root rtd unknown /proc/73634/root (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73634 root txt unknown /proc/73634/exe (readlink: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 73634 root NOFD /proc/73634/fd (opendir: Permission denied) 2024-03-16T11:08:29.771 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73636 root cwd unknown /proc/73636/cwd (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73636 root rtd unknown /proc/73636/root (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73636 root txt unknown /proc/73636/exe (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 73636 root NOFD /proc/73636/fd (opendir: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 ceph cwd unknown /proc/73640/cwd (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 ceph rtd unknown /proc/73640/root (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 ceph txt unknown /proc/73640/exe (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 ceph NOFD /proc/73640/fd (opendir: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73641 log ceph cwd unknown /proc/73640/task/73641/cwd (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73641 log ceph rtd unknown /proc/73640/task/73641/root (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73641 log ceph txt unknown /proc/73640/task/73641/exe (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73641 log ceph NOFD /proc/73640/task/73641/fd (opendir: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73643 msgr-work ceph cwd unknown /proc/73640/task/73643/cwd (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73643 msgr-work ceph rtd unknown /proc/73640/task/73643/root (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73643 msgr-work ceph txt unknown /proc/73640/task/73643/exe (readlink: Permission denied) 2024-03-16T11:08:29.772 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73643 msgr-work ceph NOFD /proc/73640/task/73643/fd (opendir: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73644 msgr-work ceph cwd unknown /proc/73640/task/73644/cwd (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73644 msgr-work ceph rtd unknown /proc/73640/task/73644/root (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73644 msgr-work ceph txt unknown /proc/73640/task/73644/exe (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73644 msgr-work ceph NOFD /proc/73640/task/73644/fd (opendir: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73645 msgr-work ceph cwd unknown /proc/73640/task/73645/cwd (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73645 msgr-work ceph rtd unknown /proc/73640/task/73645/root (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73645 msgr-work ceph txt unknown /proc/73640/task/73645/exe (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73645 msgr-work ceph NOFD /proc/73640/task/73645/fd (opendir: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73649 service ceph cwd unknown /proc/73640/task/73649/cwd (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73649 service ceph rtd unknown /proc/73640/task/73649/root (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73649 service ceph txt unknown /proc/73640/task/73649/exe (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73649 service ceph NOFD /proc/73640/task/73649/fd (opendir: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73650 admin_soc ceph cwd unknown /proc/73640/task/73650/cwd (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73650 admin_soc ceph rtd unknown /proc/73640/task/73650/root (readlink: Permission denied) 2024-03-16T11:08:29.773 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73650 admin_soc ceph txt unknown /proc/73640/task/73650/exe (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73650 admin_soc ceph NOFD /proc/73640/task/73650/fd (opendir: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73651 signal_ha ceph cwd unknown /proc/73640/task/73651/cwd (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73651 signal_ha ceph rtd unknown /proc/73640/task/73651/root (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73651 signal_ha ceph txt unknown /proc/73640/task/73651/exe (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73651 signal_ha ceph NOFD /proc/73640/task/73651/fd (opendir: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73652 io_contex ceph cwd unknown /proc/73640/task/73652/cwd (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73652 io_contex ceph rtd unknown /proc/73640/task/73652/root (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73652 io_contex ceph txt unknown /proc/73640/task/73652/exe (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73652 io_contex ceph NOFD /proc/73640/task/73652/fd (opendir: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73653 io_contex ceph cwd unknown /proc/73640/task/73653/cwd (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73653 io_contex ceph rtd unknown /proc/73640/task/73653/root (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73653 io_contex ceph txt unknown /proc/73640/task/73653/exe (readlink: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73653 io_contex ceph NOFD /proc/73640/task/73653/fd (opendir: Permission denied) 2024-03-16T11:08:29.774 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73654 ms_dispat ceph cwd unknown /proc/73640/task/73654/cwd (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73654 ms_dispat ceph rtd unknown /proc/73640/task/73654/root (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73654 ms_dispat ceph txt unknown /proc/73640/task/73654/exe (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73654 ms_dispat ceph NOFD /proc/73640/task/73654/fd (opendir: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73655 ms_local ceph cwd unknown /proc/73640/task/73655/cwd (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73655 ms_local ceph rtd unknown /proc/73640/task/73655/root (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73655 ms_local ceph txt unknown /proc/73640/task/73655/exe (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73655 ms_local ceph NOFD /proc/73640/task/73655/fd (opendir: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73656 safe_time ceph cwd unknown /proc/73640/task/73656/cwd (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73656 safe_time ceph rtd unknown /proc/73640/task/73656/root (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73656 safe_time ceph txt unknown /proc/73640/task/73656/exe (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73656 safe_time ceph NOFD /proc/73640/task/73656/fd (opendir: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73657 safe_time ceph cwd unknown /proc/73640/task/73657/cwd (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73657 safe_time ceph rtd unknown /proc/73640/task/73657/root (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73657 safe_time ceph txt unknown /proc/73640/task/73657/exe (readlink: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73657 safe_time ceph NOFD /proc/73640/task/73657/fd (opendir: Permission denied) 2024-03-16T11:08:29.775 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73658 ceph-mds ceph cwd unknown /proc/73640/task/73658/cwd (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73658 ceph-mds ceph rtd unknown /proc/73640/task/73658/root (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73658 ceph-mds ceph txt unknown /proc/73640/task/73658/exe (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73658 ceph-mds ceph NOFD /proc/73640/task/73658/fd (opendir: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73663 safe_time ceph cwd unknown /proc/73640/task/73663/cwd (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73663 safe_time ceph rtd unknown /proc/73640/task/73663/root (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73663 safe_time ceph txt unknown /proc/73640/task/73663/exe (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 73640 73663 safe_time ceph NOFD /proc/73640/task/73663/fd (opendir: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74170 root cwd unknown /proc/74170/cwd (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74170 root rtd unknown /proc/74170/root (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74170 root txt unknown /proc/74170/exe (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74170 root NOFD /proc/74170/fd (opendir: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74171 root cwd unknown /proc/74171/cwd (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74171 root rtd unknown /proc/74171/root (readlink: Permission denied) 2024-03-16T11:08:29.776 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74171 root txt unknown /proc/74171/exe (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/6 74171 root NOFD /proc/74171/fd (opendir: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74236 root cwd unknown /proc/74236/cwd (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74236 root rtd unknown /proc/74236/root (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74236 root txt unknown /proc/74236/exe (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74236 root NOFD /proc/74236/fd (opendir: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74238 root cwd unknown /proc/74238/cwd (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74238 root rtd unknown /proc/74238/root (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74238 root txt unknown /proc/74238/exe (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74238 root NOFD /proc/74238/fd (opendir: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 ceph cwd unknown /proc/74240/cwd (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 ceph rtd unknown /proc/74240/root (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 ceph txt unknown /proc/74240/exe (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 ceph NOFD /proc/74240/fd (opendir: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74241 log ceph cwd unknown /proc/74240/task/74241/cwd (readlink: Permission denied) 2024-03-16T11:08:29.777 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74241 log ceph rtd unknown /proc/74240/task/74241/root (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74241 log ceph txt unknown /proc/74240/task/74241/exe (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74241 log ceph NOFD /proc/74240/task/74241/fd (opendir: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74243 msgr-work ceph cwd unknown /proc/74240/task/74243/cwd (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74243 msgr-work ceph rtd unknown /proc/74240/task/74243/root (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74243 msgr-work ceph txt unknown /proc/74240/task/74243/exe (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74243 msgr-work ceph NOFD /proc/74240/task/74243/fd (opendir: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74244 msgr-work ceph cwd unknown /proc/74240/task/74244/cwd (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74244 msgr-work ceph rtd unknown /proc/74240/task/74244/root (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74244 msgr-work ceph txt unknown /proc/74240/task/74244/exe (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74244 msgr-work ceph NOFD /proc/74240/task/74244/fd (opendir: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74245 msgr-work ceph cwd unknown /proc/74240/task/74245/cwd (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74245 msgr-work ceph rtd unknown /proc/74240/task/74245/root (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74245 msgr-work ceph txt unknown /proc/74240/task/74245/exe (readlink: Permission denied) 2024-03-16T11:08:29.778 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74245 msgr-work ceph NOFD /proc/74240/task/74245/fd (opendir: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74249 service ceph cwd unknown /proc/74240/task/74249/cwd (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74249 service ceph rtd unknown /proc/74240/task/74249/root (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74249 service ceph txt unknown /proc/74240/task/74249/exe (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74249 service ceph NOFD /proc/74240/task/74249/fd (opendir: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74250 admin_soc ceph cwd unknown /proc/74240/task/74250/cwd (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74250 admin_soc ceph rtd unknown /proc/74240/task/74250/root (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74250 admin_soc ceph txt unknown /proc/74240/task/74250/exe (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74250 admin_soc ceph NOFD /proc/74240/task/74250/fd (opendir: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74251 signal_ha ceph cwd unknown /proc/74240/task/74251/cwd (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74251 signal_ha ceph rtd unknown /proc/74240/task/74251/root (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74251 signal_ha ceph txt unknown /proc/74240/task/74251/exe (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74251 signal_ha ceph NOFD /proc/74240/task/74251/fd (opendir: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74252 io_contex ceph cwd unknown /proc/74240/task/74252/cwd (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74252 io_contex ceph rtd unknown /proc/74240/task/74252/root (readlink: Permission denied) 2024-03-16T11:08:29.779 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74252 io_contex ceph txt unknown /proc/74240/task/74252/exe (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74252 io_contex ceph NOFD /proc/74240/task/74252/fd (opendir: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74253 io_contex ceph cwd unknown /proc/74240/task/74253/cwd (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74253 io_contex ceph rtd unknown /proc/74240/task/74253/root (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74253 io_contex ceph txt unknown /proc/74240/task/74253/exe (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74253 io_contex ceph NOFD /proc/74240/task/74253/fd (opendir: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74254 ms_dispat ceph cwd unknown /proc/74240/task/74254/cwd (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74254 ms_dispat ceph rtd unknown /proc/74240/task/74254/root (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74254 ms_dispat ceph txt unknown /proc/74240/task/74254/exe (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74254 ms_dispat ceph NOFD /proc/74240/task/74254/fd (opendir: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74255 ms_local ceph cwd unknown /proc/74240/task/74255/cwd (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74255 ms_local ceph rtd unknown /proc/74240/task/74255/root (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74255 ms_local ceph txt unknown /proc/74240/task/74255/exe (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74255 ms_local ceph NOFD /proc/74240/task/74255/fd (opendir: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74256 safe_time ceph cwd unknown /proc/74240/task/74256/cwd (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74256 safe_time ceph rtd unknown /proc/74240/task/74256/root (readlink: Permission denied) 2024-03-16T11:08:29.780 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74256 safe_time ceph txt unknown /proc/74240/task/74256/exe (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74256 safe_time ceph NOFD /proc/74240/task/74256/fd (opendir: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74257 safe_time ceph cwd unknown /proc/74240/task/74257/cwd (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74257 safe_time ceph rtd unknown /proc/74240/task/74257/root (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74257 safe_time ceph txt unknown /proc/74240/task/74257/exe (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74257 safe_time ceph NOFD /proc/74240/task/74257/fd (opendir: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74258 ceph-mds ceph cwd unknown /proc/74240/task/74258/cwd (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74258 ceph-mds ceph rtd unknown /proc/74240/task/74258/root (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74258 ceph-mds ceph txt unknown /proc/74240/task/74258/exe (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74258 ceph-mds ceph NOFD /proc/74240/task/74258/fd (opendir: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74372 safe_time ceph cwd unknown /proc/74240/task/74372/cwd (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74372 safe_time ceph rtd unknown /proc/74240/task/74372/root (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74372 safe_time ceph txt unknown /proc/74240/task/74372/exe (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74240 74372 safe_time ceph NOFD /proc/74240/task/74372/fd (opendir: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74596 root cwd unknown /proc/74596/cwd (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74596 root rtd unknown /proc/74596/root (readlink: Permission denied) 2024-03-16T11:08:29.781 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74596 root txt unknown /proc/74596/exe (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:conmon 74596 root NOFD /proc/74596/fd (opendir: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74598 root cwd unknown /proc/74598/cwd (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74598 root rtd unknown /proc/74598/root (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74598 root txt unknown /proc/74598/exe (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:podman-in 74598 root NOFD /proc/74598/fd (opendir: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 ceph cwd unknown /proc/74600/cwd (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 ceph rtd unknown /proc/74600/root (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 ceph txt unknown /proc/74600/exe (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 ceph NOFD /proc/74600/fd (opendir: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74601 log ceph cwd unknown /proc/74600/task/74601/cwd (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74601 log ceph rtd unknown /proc/74600/task/74601/root (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74601 log ceph txt unknown /proc/74600/task/74601/exe (readlink: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74601 log ceph NOFD /proc/74600/task/74601/fd (opendir: Permission denied) 2024-03-16T11:08:29.782 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74603 msgr-work ceph cwd unknown /proc/74600/task/74603/cwd (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74603 msgr-work ceph rtd unknown /proc/74600/task/74603/root (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74603 msgr-work ceph txt unknown /proc/74600/task/74603/exe (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74603 msgr-work ceph NOFD /proc/74600/task/74603/fd (opendir: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74604 msgr-work ceph cwd unknown /proc/74600/task/74604/cwd (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74604 msgr-work ceph rtd unknown /proc/74600/task/74604/root (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74604 msgr-work ceph txt unknown /proc/74600/task/74604/exe (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74604 msgr-work ceph NOFD /proc/74600/task/74604/fd (opendir: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74605 msgr-work ceph cwd unknown /proc/74600/task/74605/cwd (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74605 msgr-work ceph rtd unknown /proc/74600/task/74605/root (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74605 msgr-work ceph txt unknown /proc/74600/task/74605/exe (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74605 msgr-work ceph NOFD /proc/74600/task/74605/fd (opendir: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74609 service ceph cwd unknown /proc/74600/task/74609/cwd (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74609 service ceph rtd unknown /proc/74600/task/74609/root (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74609 service ceph txt unknown /proc/74600/task/74609/exe (readlink: Permission denied) 2024-03-16T11:08:29.783 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74609 service ceph NOFD /proc/74600/task/74609/fd (opendir: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74610 admin_soc ceph cwd unknown /proc/74600/task/74610/cwd (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74610 admin_soc ceph rtd unknown /proc/74600/task/74610/root (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74610 admin_soc ceph txt unknown /proc/74600/task/74610/exe (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74610 admin_soc ceph NOFD /proc/74600/task/74610/fd (opendir: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74611 signal_ha ceph cwd unknown /proc/74600/task/74611/cwd (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74611 signal_ha ceph rtd unknown /proc/74600/task/74611/root (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74611 signal_ha ceph txt unknown /proc/74600/task/74611/exe (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74611 signal_ha ceph NOFD /proc/74600/task/74611/fd (opendir: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74612 io_contex ceph cwd unknown /proc/74600/task/74612/cwd (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74612 io_contex ceph rtd unknown /proc/74600/task/74612/root (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74612 io_contex ceph txt unknown /proc/74600/task/74612/exe (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74612 io_contex ceph NOFD /proc/74600/task/74612/fd (opendir: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74613 io_contex ceph cwd unknown /proc/74600/task/74613/cwd (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74613 io_contex ceph rtd unknown /proc/74600/task/74613/root (readlink: Permission denied) 2024-03-16T11:08:29.784 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74613 io_contex ceph txt unknown /proc/74600/task/74613/exe (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74613 io_contex ceph NOFD /proc/74600/task/74613/fd (opendir: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74614 ms_dispat ceph cwd unknown /proc/74600/task/74614/cwd (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74614 ms_dispat ceph rtd unknown /proc/74600/task/74614/root (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74614 ms_dispat ceph txt unknown /proc/74600/task/74614/exe (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74614 ms_dispat ceph NOFD /proc/74600/task/74614/fd (opendir: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74615 ms_local ceph cwd unknown /proc/74600/task/74615/cwd (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74615 ms_local ceph rtd unknown /proc/74600/task/74615/root (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74615 ms_local ceph txt unknown /proc/74600/task/74615/exe (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74615 ms_local ceph NOFD /proc/74600/task/74615/fd (opendir: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74616 safe_time ceph cwd unknown /proc/74600/task/74616/cwd (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74616 safe_time ceph rtd unknown /proc/74600/task/74616/root (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74616 safe_time ceph txt unknown /proc/74600/task/74616/exe (readlink: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74616 safe_time ceph NOFD /proc/74600/task/74616/fd (opendir: Permission denied) 2024-03-16T11:08:29.785 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74617 safe_time ceph cwd unknown /proc/74600/task/74617/cwd (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74617 safe_time ceph rtd unknown /proc/74600/task/74617/root (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74617 safe_time ceph txt unknown /proc/74600/task/74617/exe (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74617 safe_time ceph NOFD /proc/74600/task/74617/fd (opendir: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74618 ceph-mds ceph cwd unknown /proc/74600/task/74618/cwd (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74618 ceph-mds ceph rtd unknown /proc/74600/task/74618/root (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74618 ceph-mds ceph txt unknown /proc/74600/task/74618/exe (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74618 ceph-mds ceph NOFD /proc/74600/task/74618/fd (opendir: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74623 safe_time ceph cwd unknown /proc/74600/task/74623/cwd (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74623 safe_time ceph rtd unknown /proc/74600/task/74623/root (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74623 safe_time ceph txt unknown /proc/74600/task/74623/exe (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-mds 74600 74623 safe_time ceph NOFD /proc/74600/task/74623/fd (opendir: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 77912 root cwd unknown /proc/77912/cwd (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 77912 root rtd unknown /proc/77912/root (readlink: Permission denied) 2024-03-16T11:08:29.786 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 77912 root txt unknown /proc/77912/exe (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/0 77912 root NOFD /proc/77912/fd (opendir: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 78295 root cwd unknown /proc/78295/cwd (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 78295 root rtd unknown /proc/78295/root (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 78295 root txt unknown /proc/78295/exe (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 78295 root NOFD /proc/78295/fd (opendir: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 78992 root cwd unknown /proc/78992/cwd (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 78992 root rtd unknown /proc/78992/root (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 78992 root txt unknown /proc/78992/exe (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 78992 root NOFD /proc/78992/fd (opendir: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 80739 root cwd unknown /proc/80739/cwd (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 80739 root rtd unknown /proc/80739/root (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 80739 root txt unknown /proc/80739/exe (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/7 80739 root NOFD /proc/80739/fd (opendir: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 81370 root cwd unknown /proc/81370/cwd (readlink: Permission denied) 2024-03-16T11:08:29.787 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 81370 root rtd unknown /proc/81370/root (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 81370 root txt unknown /proc/81370/exe (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/1 81370 root NOFD /proc/81370/fd (opendir: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 81659 root cwd unknown /proc/81659/cwd (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 81659 root rtd unknown /proc/81659/root (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 81659 root txt unknown /proc/81659/exe (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/4 81659 root NOFD /proc/81659/fd (opendir: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 84564 root cwd unknown /proc/84564/cwd (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 84564 root rtd unknown /proc/84564/root (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 84564 root txt unknown /proc/84564/exe (readlink: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/3 84564 root NOFD /proc/84564/fd (opendir: Permission denied) 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu cwd DIR 8,1 4096 1026 /home/ubuntu 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu txt REG 8,1 1388920 9667 /usr/bin/bash 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-16T11:08:29.788 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 195080 9664 /usr/lib64/libtinfo.so.6.2 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T11:08:29.789 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu 0r FIFO 0,13 0t0 222832 pipe 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu 1w FIFO 0,13 0t0 222833 pipe 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:bash 84700 ubuntu 2w FIFO 0,13 0t0 222834 pipe 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84724 root cwd unknown /proc/84724/cwd (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84724 root rtd unknown /proc/84724/root (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84724 root txt unknown /proc/84724/exe (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84724 root NOFD /proc/84724/fd (opendir: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84735 root cwd unknown /proc/84735/cwd (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84735 root rtd unknown /proc/84735/root (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84735 root txt unknown /proc/84735/exe (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 84735 root NOFD /proc/84735/fd (opendir: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 84738 root cwd unknown /proc/84738/cwd (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 84738 root rtd unknown /proc/84738/root (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 84738 root txt unknown /proc/84738/exe (readlink: Permission denied) 2024-03-16T11:08:29.790 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 84738 root NOFD /proc/84738/fd (opendir: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 root cwd unknown /proc/84740/cwd (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 root rtd unknown /proc/84740/root (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 root txt unknown /proc/84740/exe (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 root NOFD /proc/84740/fd (opendir: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84759 log root cwd unknown /proc/84740/task/84759/cwd (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84759 log root rtd unknown /proc/84740/task/84759/root (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84759 log root txt unknown /proc/84740/task/84759/exe (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84759 log root NOFD /proc/84740/task/84759/fd (opendir: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84761 msgr-work root cwd unknown /proc/84740/task/84761/cwd (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84761 msgr-work root rtd unknown /proc/84740/task/84761/root (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84761 msgr-work root txt unknown /proc/84740/task/84761/exe (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84761 msgr-work root NOFD /proc/84740/task/84761/fd (opendir: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84762 msgr-work root cwd unknown /proc/84740/task/84762/cwd (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84762 msgr-work root rtd unknown /proc/84740/task/84762/root (readlink: Permission denied) 2024-03-16T11:08:29.791 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84762 msgr-work root txt unknown /proc/84740/task/84762/exe (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84762 msgr-work root NOFD /proc/84740/task/84762/fd (opendir: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84763 msgr-work root cwd unknown /proc/84740/task/84763/cwd (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84763 msgr-work root rtd unknown /proc/84740/task/84763/root (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84763 msgr-work root txt unknown /proc/84740/task/84763/exe (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84763 msgr-work root NOFD /proc/84740/task/84763/fd (opendir: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84767 service root cwd unknown /proc/84740/task/84767/cwd (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84767 service root rtd unknown /proc/84740/task/84767/root (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84767 service root txt unknown /proc/84740/task/84767/exe (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84767 service root NOFD /proc/84740/task/84767/fd (opendir: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84768 admin_soc root cwd unknown /proc/84740/task/84768/cwd (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84768 admin_soc root rtd unknown /proc/84740/task/84768/root (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84768 admin_soc root txt unknown /proc/84740/task/84768/exe (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84768 admin_soc root NOFD /proc/84740/task/84768/fd (opendir: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84769 signal_ha root cwd unknown /proc/84740/task/84769/cwd (readlink: Permission denied) 2024-03-16T11:08:29.792 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84769 signal_ha root rtd unknown /proc/84740/task/84769/root (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84769 signal_ha root txt unknown /proc/84740/task/84769/exe (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84769 signal_ha root NOFD /proc/84740/task/84769/fd (opendir: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84770 io_contex root cwd unknown /proc/84740/task/84770/cwd (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84770 io_contex root rtd unknown /proc/84740/task/84770/root (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84770 io_contex root txt unknown /proc/84740/task/84770/exe (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84770 io_contex root NOFD /proc/84740/task/84770/fd (opendir: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84771 io_contex root cwd unknown /proc/84740/task/84771/cwd (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84771 io_contex root rtd unknown /proc/84740/task/84771/root (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84771 io_contex root txt unknown /proc/84740/task/84771/exe (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84771 io_contex root NOFD /proc/84740/task/84771/fd (opendir: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84772 ceph_time root cwd unknown /proc/84740/task/84772/cwd (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84772 ceph_time root rtd unknown /proc/84740/task/84772/root (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84772 ceph_time root txt unknown /proc/84740/task/84772/exe (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84772 ceph_time root NOFD /proc/84740/task/84772/fd (opendir: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84773 fn_anonym root cwd unknown /proc/84740/task/84773/cwd (readlink: Permission denied) 2024-03-16T11:08:29.793 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84773 fn_anonym root rtd unknown /proc/84740/task/84773/root (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84773 fn_anonym root txt unknown /proc/84740/task/84773/exe (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84773 fn_anonym root NOFD /proc/84740/task/84773/fd (opendir: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84774 fn_anonym root cwd unknown /proc/84740/task/84774/cwd (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84774 fn_anonym root rtd unknown /proc/84740/task/84774/root (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84774 fn_anonym root txt unknown /proc/84740/task/84774/exe (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84774 fn_anonym root NOFD /proc/84740/task/84774/fd (opendir: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84775 fn_anonym root cwd unknown /proc/84740/task/84775/cwd (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84775 fn_anonym root rtd unknown /proc/84740/task/84775/root (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84775 fn_anonym root txt unknown /proc/84740/task/84775/exe (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84775 fn_anonym root NOFD /proc/84740/task/84775/fd (opendir: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84776 fn_anonym root cwd unknown /proc/84740/task/84776/cwd (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84776 fn_anonym root rtd unknown /proc/84740/task/84776/root (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84776 fn_anonym root txt unknown /proc/84740/task/84776/exe (readlink: Permission denied) 2024-03-16T11:08:29.794 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84776 fn_anonym root NOFD /proc/84740/task/84776/fd (opendir: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84777 fn_anonym root cwd unknown /proc/84740/task/84777/cwd (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84777 fn_anonym root rtd unknown /proc/84740/task/84777/root (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84777 fn_anonym root txt unknown /proc/84740/task/84777/exe (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84777 fn_anonym root NOFD /proc/84740/task/84777/fd (opendir: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84778 safe_time root cwd unknown /proc/84740/task/84778/cwd (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84778 safe_time root rtd unknown /proc/84740/task/84778/root (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84778 safe_time root txt unknown /proc/84740/task/84778/exe (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84778 safe_time root NOFD /proc/84740/task/84778/fd (opendir: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84779 fn_anonym root cwd unknown /proc/84740/task/84779/cwd (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84779 fn_anonym root rtd unknown /proc/84740/task/84779/root (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84779 fn_anonym root txt unknown /proc/84740/task/84779/exe (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84779 fn_anonym root NOFD /proc/84740/task/84779/fd (opendir: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84780 flusher root cwd unknown /proc/84740/task/84780/cwd (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84780 flusher root rtd unknown /proc/84740/task/84780/root (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84780 flusher root txt unknown /proc/84740/task/84780/exe (readlink: Permission denied) 2024-03-16T11:08:29.795 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84780 flusher root NOFD /proc/84740/task/84780/fd (opendir: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84781 ms_dispat root cwd unknown /proc/84740/task/84781/cwd (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84781 ms_dispat root rtd unknown /proc/84740/task/84781/root (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84781 ms_dispat root txt unknown /proc/84740/task/84781/exe (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84781 ms_dispat root NOFD /proc/84740/task/84781/fd (opendir: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84782 ms_local root cwd unknown /proc/84740/task/84782/cwd (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84782 ms_local root rtd unknown /proc/84740/task/84782/root (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84782 ms_local root txt unknown /proc/84740/task/84782/exe (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84782 ms_local root NOFD /proc/84740/task/84782/fd (opendir: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84783 safe_time root cwd unknown /proc/84740/task/84783/cwd (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84783 safe_time root rtd unknown /proc/84740/task/84783/root (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84783 safe_time root txt unknown /proc/84740/task/84783/exe (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84783 safe_time root NOFD /proc/84740/task/84783/fd (opendir: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84784 ceph-fuse root cwd unknown /proc/84740/task/84784/cwd (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84784 ceph-fuse root rtd unknown /proc/84740/task/84784/root (readlink: Permission denied) 2024-03-16T11:08:29.796 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84784 ceph-fuse root txt unknown /proc/84740/task/84784/exe (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84784 ceph-fuse root NOFD /proc/84740/task/84784/fd (opendir: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84800 ceph-fuse root cwd unknown /proc/84740/task/84800/cwd (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84800 ceph-fuse root rtd unknown /proc/84740/task/84800/root (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84800 ceph-fuse root txt unknown /proc/84740/task/84800/exe (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84800 ceph-fuse root NOFD /proc/84740/task/84800/fd (opendir: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84802 ceph-fuse root cwd unknown /proc/84740/task/84802/cwd (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84802 ceph-fuse root rtd unknown /proc/84740/task/84802/root (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84802 ceph-fuse root txt unknown /proc/84740/task/84802/exe (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 84802 ceph-fuse root NOFD /proc/84740/task/84802/fd (opendir: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87892 ceph-fuse root cwd unknown /proc/84740/task/87892/cwd (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87892 ceph-fuse root rtd unknown /proc/84740/task/87892/root (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87892 ceph-fuse root txt unknown /proc/84740/task/87892/exe (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87892 ceph-fuse root NOFD /proc/84740/task/87892/fd (opendir: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87900 ceph-fuse root cwd unknown /proc/84740/task/87900/cwd (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87900 ceph-fuse root rtd unknown /proc/84740/task/87900/root (readlink: Permission denied) 2024-03-16T11:08:29.797 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87900 ceph-fuse root txt unknown /proc/84740/task/87900/exe (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 87900 ceph-fuse root NOFD /proc/84740/task/87900/fd (opendir: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88052 ceph-fuse root cwd unknown /proc/84740/task/88052/cwd (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88052 ceph-fuse root rtd unknown /proc/84740/task/88052/root (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88052 ceph-fuse root txt unknown /proc/84740/task/88052/exe (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88052 ceph-fuse root NOFD /proc/84740/task/88052/fd (opendir: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88578 ceph-fuse root cwd unknown /proc/84740/task/88578/cwd (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88578 ceph-fuse root rtd unknown /proc/84740/task/88578/root (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88578 ceph-fuse root txt unknown /proc/84740/task/88578/exe (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 88578 ceph-fuse root NOFD /proc/84740/task/88578/fd (opendir: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 89019 ceph-fuse root cwd unknown /proc/84740/task/89019/cwd (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 89019 ceph-fuse root rtd unknown /proc/84740/task/89019/root (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 89019 ceph-fuse root txt unknown /proc/84740/task/89019/exe (readlink: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:ceph-fuse 84740 89019 ceph-fuse root NOFD /proc/84740/task/89019/fd (opendir: Permission denied) 2024-03-16T11:08:29.798 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 85707 root cwd unknown /proc/85707/cwd (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 85707 root rtd unknown /proc/85707/root (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 85707 root txt unknown /proc/85707/exe (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:sudo 85707 root NOFD /proc/85707/fd (opendir: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 85745 root cwd unknown /proc/85745/cwd (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 85745 root rtd unknown /proc/85745/root (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 85745 root txt unknown /proc/85745/exe (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:daemon-he 85745 root NOFD /proc/85745/fd (opendir: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:bash 85761 root cwd unknown /proc/85761/cwd (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:bash 85761 root rtd unknown /proc/85761/root (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:bash 85761 root txt unknown /proc/85761/exe (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:bash 85761 root NOFD /proc/85761/fd (opendir: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 86445 root cwd unknown /proc/86445/cwd (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 86445 root rtd unknown /proc/86445/root (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 86445 root txt unknown /proc/86445/exe (readlink: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:kworker/2 86445 root NOFD /proc/86445/fd (opendir: Permission denied) 2024-03-16T11:08:29.799 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu cwd DIR 0,119 194248488 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu txt REG 8,1 40944 10098 /usr/bin/timeout 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-16T11:08:29.800 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu 0r FIFO 0,13 0t0 227288 pipe 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu 1w FIFO 0,13 0t0 227289 pipe 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:timeout 87872 ubuntu 2w FIFO 0,13 0t0 227290 pipe 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu cwd DIR 0,119 194248488 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu txt REG 8,1 1388920 9667 /usr/bin/bash 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-16T11:08:29.801 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 195080 9664 /usr/lib64/libtinfo.so.6.2 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu 0r FIFO 0,13 0t0 227288 pipe 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu 1w FIFO 0,13 0t0 227289 pipe 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu 2w FIFO 0,13 0t0 227290 pipe 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:fsync-tes 87890 ubuntu 255r REG 8,1 207 530694 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:sleep 89127 root cwd unknown /proc/89127/cwd (readlink: Permission denied) 2024-03-16T11:08:29.802 INFO:tasks.workunit.client.0.smithi078.stdout:sleep 89127 root rtd unknown /proc/89127/root (readlink: Permission denied) 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:sleep 89127 root txt unknown /proc/89127/exe (readlink: Permission denied) 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:sleep 89127 root NOFD /proc/89127/fd (opendir: Permission denied) 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu cwd DIR 0,119 194248488 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 4487176 2249 /usr/lib64/libcrypto.so.3.0.7 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-16T11:08:29.803 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu 0r FIFO 0,13 0t0 227288 pipe 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu 1w FIFO 0,13 0t0 227289 pipe 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu 2w FIFO 0,13 0t0 227290 pipe 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu 3r DIR 0,20 0 1 /proc 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu 4r DIR 0,20 0 232077 /proc/89129/fd 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu 5w FIFO 0,13 0t0 232082 pipe 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89129 ubuntu 6r FIFO 0,13 0t0 232083 pipe 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu cwd DIR 0,119 194248488 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T11:08:29.804 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 4487176 2249 /usr/lib64/libcrypto.so.3.0.7 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-16T11:08:29.805 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-16T11:08:29.806 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-16T11:08:29.806 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T11:08:29.806 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T11:08:29.806 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu 4r FIFO 0,13 0t0 232082 pipe 2024-03-16T11:08:29.806 INFO:tasks.workunit.client.0.smithi078.stdout:lsof 89130 ubuntu 7w FIFO 0,13 0t0 232083 pipe 2024-03-16T11:08:29.806 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-16T11:08:29.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T11:08:29.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:29 smithi078 ceph-mon[32235]: pgmap v423: 97 pgs: 97 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 18 MiB/s wr, 4.09k op/s 2024-03-16T11:08:29.870 INFO:tasks.workunit:Stopping ['suites/fsync-tester.sh'] on client.0... 2024-03-16T11:08:29.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-16T11:08:30.137 DEBUG:teuthology.parallel:result is None 2024-03-16T11:08:30.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T11:08:30.171 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T11:08:30.171 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-16T11:08:30.181 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T11:08:30.219 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-16T11:08:30.228 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-16T11:08:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:31 smithi098 ceph-mon[35848]: pgmap v424: 97 pgs: 97 active+clean; 258 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:31 smithi153 ceph-mon[35521]: pgmap v424: 97 pgs: 97 active+clean; 258 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:31 smithi078 ceph-mon[32235]: pgmap v424: 97 pgs: 97 active+clean; 258 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 18 MiB/s wr, 4.17k op/s 2024-03-16T11:08:33.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:33 smithi098 ceph-mon[35848]: pgmap v425: 97 pgs: 97 active+clean; 258 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2024-03-16T11:08:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:33 smithi153 ceph-mon[35521]: pgmap v425: 97 pgs: 97 active+clean; 258 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2024-03-16T11:08:33.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:33 smithi078 ceph-mon[32235]: pgmap v425: 97 pgs: 97 active+clean; 258 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2024-03-16T11:08:35.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:35 smithi098 ceph-mon[35848]: pgmap v426: 97 pgs: 97 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 16 MiB/s wr, 3.69k op/s 2024-03-16T11:08:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:35 smithi153 ceph-mon[35521]: pgmap v426: 97 pgs: 97 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 16 MiB/s wr, 3.69k op/s 2024-03-16T11:08:35.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:35 smithi078 ceph-mon[32235]: pgmap v426: 97 pgs: 97 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 16 MiB/s wr, 3.69k op/s 2024-03-16T11:08:37.361 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:37 smithi078 ceph-mon[32235]: pgmap v427: 97 pgs: 97 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 3.10k op/s 2024-03-16T11:08:37.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:37 smithi098 ceph-mon[35848]: pgmap v427: 97 pgs: 97 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 3.10k op/s 2024-03-16T11:08:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:37 smithi153 ceph-mon[35521]: pgmap v427: 97 pgs: 97 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 3.10k op/s 2024-03-16T11:08:39.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:39 smithi098 ceph-mon[35848]: pgmap v428: 97 pgs: 97 active+clean; 142 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 9.7 MiB/s wr, 2.22k op/s 2024-03-16T11:08:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:39 smithi153 ceph-mon[35521]: pgmap v428: 97 pgs: 97 active+clean; 142 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 9.7 MiB/s wr, 2.22k op/s 2024-03-16T11:08:39.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:39 smithi078 ceph-mon[32235]: pgmap v428: 97 pgs: 97 active+clean; 142 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 9.7 MiB/s wr, 2.22k op/s 2024-03-16T11:08:41.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T11:08:41.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T11:08:41.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 -- 172.21.15.78:0/1724127994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 msgr2=0x7f4454102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1724127994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f4454102140 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4444009900 tx=0x7f444402bc30 comp rx=0 tx=0).stop 2024-03-16T11:08:41.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 -- 172.21.15.78:0/1724127994 shutdown_connections 2024-03-16T11:08:41.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1724127994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4454101330 0x7f4454108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1724127994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4454100970 0x7f4454100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1724127994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f4454102140 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 -- 172.21.15.78:0/1724127994 >> 172.21.15.78:0/1724127994 conn(0x7f44540fb3e0 msgr2=0x7f44540fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 -- 172.21.15.78:0/1724127994 shutdown_connections 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.261+0000 7f445a7d4640 1 -- 172.21.15.78:0/1724127994 wait complete. 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 Processor -- start 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 -- start start 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4454100970 0x7f4454197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4454101330 0x7f4454197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f445419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f445410d3c0 con 0x7f4454101d20 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f445410d240 con 0x7f4454100970 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f445a7d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f445410d540 con 0x7f4454101330 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f445419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4453fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4454100970 0x7f4454197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f445419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52846/0 (socket says 172.21.15.78:52846) 2024-03-16T11:08:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 -- 172.21.15.78:0/1789652335 learned_addr learned my addr 172.21.15.78:0/1789652335 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f44537fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4454101330 0x7f4454197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 -- 172.21.15.78:0/1789652335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4454101330 msgr2=0x7f4454197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4454101330 0x7f4454197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 -- 172.21.15.78:0/1789652335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4454100970 msgr2=0x7f4454197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4454100970 0x7f4454197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4458d71640 1 -- 172.21.15.78:0/1789652335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445419f6d0 con 0x7f4454101d20 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f4453fff640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4454100970 0x7f4454197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.262+0000 7f44537fe640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4454101330 0x7f4454197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.263+0000 7f4458d71640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f445419f060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f444c00eb10 tx=0x7f444c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.263+0000 7f44517fa640 1 -- 172.21.15.78:0/1789652335 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c004b20 con 0x7f4454101d20 2024-03-16T11:08:41.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.263+0000 7f44517fa640 1 -- 172.21.15.78:0/1789652335 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f444c004cc0 con 0x7f4454101d20 2024-03-16T11:08:41.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.263+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445419f9c0 con 0x7f4454101d20 2024-03-16T11:08:41.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.263+0000 7f44517fa640 1 -- 172.21.15.78:0/1789652335 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c010690 con 0x7f4454101d20 2024-03-16T11:08:41.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.263+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445419ff10 con 0x7f4454101d20 2024-03-16T11:08:41.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f44517fa640 1 -- 172.21.15.78:0/1789652335 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f444c005020 con 0x7f4454101d20 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 Processor -- start 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 -- start start 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb947590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb94c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb94ccd0 0x5631cb94f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5631cb8ac500 con 0x5631cb94ccd0 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5631cb854be0 con 0x5631cb947170 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f445d3be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5631cb8a85a0 con 0x5631cb947b60 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f44517fa640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f443c0a2800 0x7f443c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f44537fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb94ccd0 0x5631cb94f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb94c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f44537fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb947590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb94c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:49798/0 (socket says 172.21.15.78:49798) 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 -- 172.21.15.78:0/3470286507 learned_addr learned my addr 172.21.15.78:0/3470286507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4458d71640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f443c0a2800 0x7f443c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f44517fa640 1 -- 172.21.15.78:0/1789652335 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f444c0cf0a0 con 0x7f4454101d20 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 -- 172.21.15.78:0/3470286507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 msgr2=0x5631cb947590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 --2- 172.21.15.78:0/3470286507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb947590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 -- 172.21.15.78:0/3470286507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb94ccd0 msgr2=0x5631cb94f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 --2- 172.21.15.78:0/3470286507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb94ccd0 0x5631cb94f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.266+0000 7f4453fff640 1 -- 172.21.15.78:0/3470286507 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cb8684b0 con 0x5631cb947b60 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.267+0000 7f4453fff640 1 --2- 172.21.15.78:0/3470286507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb94c660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4444009900 tx=0x7f4444005bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac1d0d511e390a26 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.267+0000 7f4458d71640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f443c0a2800 0x7f443c0a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f444c098410 tx=0x7f444c090f20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.267+0000 7f443a7fc640 1 -- 172.21.15.78:0/3470286507 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444045020 con 0x5631cb947b60 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.267+0000 7f443a7fc640 1 -- 172.21.15.78:0/3470286507 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4444005040 con 0x5631cb947b60 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.267+0000 7f443a7fc640 1 -- 172.21.15.78:0/3470286507 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444004900 con 0x5631cb947b60 2024-03-16T11:08:41.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.267+0000 7f445d3be740 1 -- 172.21.15.78:0/3470286507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 msgr2=0x5631cb94c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.267+0000 7f445d3be740 1 --2- 172.21.15.78:0/3470286507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb94c660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4444009900 tx=0x7f4444005bf0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- 172.21.15.78:0/3470286507 shutdown_connections 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 --2- 172.21.15.78:0/3470286507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb94ccd0 0x5631cb94f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 --2- 172.21.15.78:0/3470286507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb94c660 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 --2- 172.21.15.78:0/3470286507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb947590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- 172.21.15.78:0/3470286507 >> 172.21.15.78:0/3470286507 conn(0x5631cb9426e0 msgr2=0x5631cb942ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- 172.21.15.78:0/3470286507 shutdown_connections 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- 172.21.15.78:0/3470286507 wait complete. 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- start start 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 Processor -- start 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947170 0x5631cb951640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb951b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb958bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5631cb94fc20 con 0x5631cb947170 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5631cb868160 con 0x5631cb94ccd0 2024-03-16T11:08:41.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5631cb94fda0 con 0x5631cb947b60 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f445d3be740 10 client.0 _ll_register_callbacks cb 0x5631cb84c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f44537fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947170 0x5631cb951640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f44537fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947170 0x5631cb951640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52854/0 (socket says 172.21.15.78:52854) 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.268+0000 7f44537fe640 1 -- 172.21.15.78:0/1925716449 learned_addr learned my addr 172.21.15.78:0/1925716449 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f4453fff640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb951b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f44537fe640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb958bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f4453fff640 1 -- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 msgr2=0x5631cb958bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f4453fff640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb958bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f4453fff640 1 -- 172.21.15.78:0/1925716449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947170 msgr2=0x5631cb951640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f4453fff640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947170 0x5631cb951640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f4453fff640 1 -- 172.21.15.78:0/1925716449 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cb68a7d0 con 0x5631cb947b60 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f44537fe640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947170 0x5631cb951640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f4453fff640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb951b80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5631cb943650 tx=0x7f44440048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5ba5ed6666afbb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f44337fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f44337fe640 1 -- 172.21.15.78:0/1925716449 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444050070 con 0x5631cb947b60 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f44337fe640 1 -- 172.21.15.78:0/1925716449 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f4444041480 con 0x5631cb947b60 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4444005040 con 0x5631cb947b60 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f44337fe640 1 -- 172.21.15.78:0/1925716449 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f444405b050 con 0x5631cb947b60 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f445d3be740 10 client.25912 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.269+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5631cb5c03f0 con 0x5631cb947b60 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f44337fe640 1 -- 172.21.15.78:0/1925716449 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4444005040 con 0x5631cb947b60 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f44337fe640 1 -- 172.21.15.78:0/1925716449 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f444404a450 con 0x5631cb947b60 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f445d3be740 10 client.25912 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f445d3be740 10 client.25912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f445d3be740 20 client.25912 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:41.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f445d3be740 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb95cd20 0x5631cb95f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f445d3be740 4 client.25912 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631cb8b9a40 con 0x5631cb95cd20 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f44537fe640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb95cd20 0x5631cb95f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f44537fe640 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb95cd20 0x5631cb95f110 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8aa6018907021c9 server_cookie=ba434bf347545b23 in_seq=0 out_seq=0 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.270+0000 7f44337fe640 10 client.25912 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f44337fe640 1 -- 172.21.15.78:0/1925716449 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5631cb8b9a40 con 0x5631cb95cd20 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f44337fe640 10 client.25912 handle_command_reply: tid=0 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 client.25912 shutdown 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 msgr2=0x5631cb951b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb951b80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5631cb943650 tx=0x7f44440048f0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 shutdown_connections 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb95cd20 0x5631cb95f110 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb958bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f44337fe640 0 client.25912 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:41.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947b60 0x5631cb951b80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 --2- 172.21.15.78:0/1925716449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947170 0x5631cb951640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 >> 172.21.15.78:0/1925716449 conn(0x5631cb9426e0 msgr2=0x5631cb941900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 shutdown_connections 2024-03-16T11:08:41.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 1 -- 172.21.15.78:0/1925716449 wait complete. 2024-03-16T11:08:41.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.271+0000 7f445d3be740 20 client.25912 trim_cache size 0 max 16384 2024-03-16T11:08:41.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.288+0000 7f445d3be740 1 Processor -- start 2024-03-16T11:08:41.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.288+0000 7f445d3be740 1 -- start start 2024-03-16T11:08:41.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f445d3be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb93d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f445d3be740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb958b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f445d3be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb94ccd0 0x5631cb941070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f445d3be740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f44541ab5b0 con 0x5631cb947b60 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f445d3be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44541a8e80 con 0x5631cb947170 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f445d3be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44541a6cb0 con 0x5631cb94ccd0 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb958b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb958b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52866/0 (socket says 172.21.15.78:52866) 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 -- 172.21.15.78:0/3465357905 learned_addr learned my addr 172.21.15.78:0/3465357905 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f44537fe640 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb93d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f44537fe640 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb94ccd0 0x5631cb941070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 -- 172.21.15.78:0/3465357905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb94ccd0 msgr2=0x5631cb941070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb94ccd0 0x5631cb941070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 -- 172.21.15.78:0/3465357905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 msgr2=0x5631cb93d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb93d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 -- 172.21.15.78:0/3465357905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cb5800f0 con 0x5631cb947b60 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f44537fe640 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb94ccd0 0x5631cb941070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4453fff640 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb958b80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f444402c170 tx=0x7f4444005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcc4a12a6f8134 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4439ffb640 1 -- 172.21.15.78:0/3465357905 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444000e60 con 0x5631cb947b60 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4439ffb640 1 -- 172.21.15.78:0/3465357905 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4444009520 con 0x5631cb947b60 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.289+0000 7f4439ffb640 1 -- 172.21.15.78:0/3465357905 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444050040 con 0x5631cb947b60 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 -- 172.21.15.78:0/3465357905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 msgr2=0x5631cb958b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb958b80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f444402c170 tx=0x7f4444005fd0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 -- 172.21.15.78:0/3465357905 shutdown_connections 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb94ccd0 0x5631cb941070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb958b80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 --2- 172.21.15.78:0/3465357905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb947170 0x5631cb93d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 -- 172.21.15.78:0/3465357905 >> 172.21.15.78:0/3465357905 conn(0x5631cb9426e0 msgr2=0x5631cb951640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 -- 172.21.15.78:0/3465357905 shutdown_connections 2024-03-16T11:08:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 -- 172.21.15.78:0/3465357905 wait complete. 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 -- start start 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.290+0000 7f445d3be740 1 Processor -- start 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947170 0x5631cb955a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb955f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb9c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f44541a7340 con 0x5631cb947b60 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44541a87b0 con 0x5631cb94ccd0 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44541a7a10 con 0x5631cb947170 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 10 client.0 _ll_register_callbacks cb 0x7f4448012680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb955f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb955f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52868/0 (socket says 172.21.15.78:52868) 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 -- 172.21.15.78:0/2660614543 learned_addr learned my addr 172.21.15.78:0/2660614543 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f44537fe640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947170 0x5631cb955a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f44537fe640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb9c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 -- 172.21.15.78:0/2660614543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947170 msgr2=0x5631cb955a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947170 0x5631cb955a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 -- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 msgr2=0x5631cb9c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb9c0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 -- 172.21.15.78:0/2660614543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cb8739d0 con 0x5631cb947b60 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f44537fe640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb9c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f4453fff640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb955f70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f444402cdd0 tx=0x7f4444005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afcc08aa3982d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f44337fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f44337fe640 1 -- 172.21.15.78:0/2660614543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444050070 con 0x5631cb947b60 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f44337fe640 1 -- 172.21.15.78:0/2660614543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f4444004e70 con 0x5631cb947b60 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4444009520 con 0x5631cb947b60 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f44337fe640 1 -- 172.21.15.78:0/2660614543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4444067050 con 0x5631cb947b60 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 10 client.16110 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.291+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5631cb9c2490 con 0x5631cb947b60 2024-03-16T11:08:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f44337fe640 1 -- 172.21.15.78:0/2660614543 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4444009520 con 0x5631cb947b60 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f44337fe640 1 -- 172.21.15.78:0/2660614543 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4444003780 con 0x5631cb947b60 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f445d3be740 10 client.16110 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f445d3be740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f445d3be740 20 client.16110 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f445d3be740 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb9c2de0 0x5631cb9c5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f445d3be740 4 client.16110 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5631cb8b3d90 con 0x5631cb9c2de0 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f44537fe640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb9c2de0 0x5631cb9c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f44337fe640 10 client.16110 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f44537fe640 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb9c2de0 0x5631cb9c5200 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae2c0050dce44a43 server_cookie=c4bfef7c970b49cb in_seq=0 out_seq=0 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f44337fe640 1 -- 172.21.15.78:0/2660614543 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5631cb8b3d90 con 0x5631cb9c2de0 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f44337fe640 10 client.16110 handle_command_reply: tid=0 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.292+0000 7f445d3be740 1 client.16110 shutdown 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 msgr2=0x5631cb955f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb955f70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f444402cdd0 tx=0x7f4444005c90 comp rx=0 tx=0).stop 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 shutdown_connections 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x5631cb9c2de0 0x5631cb9c5200 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5631cb94ccd0 0x5631cb9c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f44337fe640 0 client.16110 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5631cb947b60 0x5631cb955f70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 --2- 172.21.15.78:0/2660614543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631cb947170 0x5631cb955a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 >> 172.21.15.78:0/2660614543 conn(0x5631cb9426e0 msgr2=0x5631cb945940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 shutdown_connections 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 1 -- 172.21.15.78:0/2660614543 wait complete. 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445d3be740 20 client.16110 trim_cache size 0 max 16384 2024-03-16T11:08:41.300 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:41.301 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-16T11:08:41.301 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-16T11:08:41.301 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f443c0a2800 msgr2=0x7f443c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f443c0a2800 0x7f443c0a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f444c098410 tx=0x7f444c090f20 comp rx=0 tx=0).stop 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 msgr2=0x7f445419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f445419f060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f444c00eb10 tx=0x7f444c00c2d0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 shutdown_connections 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4454101d20 0x7f445419f060 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f443c0a2800 0x7f443c0a4cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4454101330 0x7f4454197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 --2- 172.21.15.78:0/1789652335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4454100970 0x7f4454197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 >> 172.21.15.78:0/1789652335 conn(0x7f44540fb3e0 msgr2=0x7f4454104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 shutdown_connections 2024-03-16T11:08:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.293+0000 7f445a7d4640 1 -- 172.21.15.78:0/1789652335 wait complete. 2024-03-16T11:08:41.304 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T11:08:41.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:08:41.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.406+0000 7f4f1edc6640 1 -- 172.21.15.78:0/737059203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 msgr2=0x7f4f18108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.406+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/737059203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f18108b40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009900 tx=0x7f4f0802bc30 comp rx=0 tx=0).stop 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.406+0000 7f4f1edc6640 1 -- 172.21.15.78:0/737059203 shutdown_connections 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/737059203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f18108b40 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/737059203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f181018f0 0x7f4f18101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/737059203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f18100700 0x7f4f18100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 -- 172.21.15.78:0/737059203 >> 172.21.15.78:0/737059203 conn(0x7f4f180fbc70 msgr2=0x7f4f180fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 -- 172.21.15.78:0/737059203 shutdown_connections 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 -- 172.21.15.78:0/737059203 wait complete. 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 Processor -- start 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 -- start start 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18100700 0x7f4f181979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181018f0 0x7f4f18197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f1819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1810c230 con 0x7f4f181022b0 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1810c0b0 con 0x7f4f18100700 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1edc6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1810c3b0 con 0x7f4f181018f0 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1d363640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f1819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1d363640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f1819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52890/0 (socket says 172.21.15.78:52890) 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1d363640 1 -- 172.21.15.78:0/2154307728 learned_addr learned my addr 172.21.15.78:0/2154307728 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f1cb62640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18100700 0x7f4f181979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.407+0000 7f4f17fff640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181018f0 0x7f4f18197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1d363640 1 -- 172.21.15.78:0/2154307728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181018f0 msgr2=0x7f4f18197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1d363640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181018f0 0x7f4f18197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1d363640 1 -- 172.21.15.78:0/2154307728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18100700 msgr2=0x7f4f181979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1d363640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18100700 0x7f4f181979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1d363640 1 -- 172.21.15.78:0/2154307728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1819f600 con 0x7f4f181022b0 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1cb62640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18100700 0x7f4f181979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1d363640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f1819ef90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009230 tx=0x7f4f08005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f15ffb640 1 -- 172.21.15.78:0/2154307728 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f08043020 con 0x7f4f181022b0 2024-03-16T11:08:41.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f15ffb640 1 -- 172.21.15.78:0/2154307728 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f080043d0 con 0x7f4f181022b0 2024-03-16T11:08:41.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f1819f890 con 0x7f4f181022b0 2024-03-16T11:08:41.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f15ffb640 1 -- 172.21.15.78:0/2154307728 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f08003b40 con 0x7f4f181022b0 2024-03-16T11:08:41.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.408+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f1819fde0 con 0x7f4f181022b0 2024-03-16T11:08:41.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.409+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f1819ffe0 con 0x7f4f181022b0 2024-03-16T11:08:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.410+0000 7f4f15ffb640 1 -- 172.21.15.78:0/2154307728 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f4f08046020 con 0x7f4f181022b0 2024-03-16T11:08:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.411+0000 7f4f15ffb640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4f040a2800 0x7f4f040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.411+0000 7f4f1cb62640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4f040a2800 0x7f4f040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.411+0000 7f4f15ffb640 1 -- 172.21.15.78:0/2154307728 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f4f080f0fa0 con 0x7f4f181022b0 2024-03-16T11:08:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.411+0000 7f4f1cb62640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4f040a2800 0x7f4f040a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f18101750 tx=0x7f4f10009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.414+0000 7f4f15ffb640 1 -- 172.21.15.78:0/2154307728 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f0804d020 con 0x7f4f181022b0 2024-03-16T11:08:41.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:41 smithi098 ceph-mon[35848]: pgmap v429: 97 pgs: 97 active+clean; 78 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.4 MiB/s wr, 1.66k op/s 2024-03-16T11:08:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:41 smithi153 ceph-mon[35521]: pgmap v429: 97 pgs: 97 active+clean; 78 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.4 MiB/s wr, 1.66k op/s 2024-03-16T11:08:41.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:41 smithi078 ceph-mon[32235]: pgmap v429: 97 pgs: 97 active+clean; 78 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.4 MiB/s wr, 1.66k op/s 2024-03-16T11:08:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.625+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f181a0370 con 0x7f4f181022b0 2024-03-16T11:08:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.626+0000 7f4f15ffb640 1 -- 172.21.15.78:0/2154307728 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7f4f080be0e0 con 0x7f4f181022b0 2024-03-16T11:08:41.627 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:08:41.627 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:08:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4f040a2800 msgr2=0x7f4f040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4f040a2800 0x7f4f040a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f18101750 tx=0x7f4f10009500 comp rx=0 tx=0).stop 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 msgr2=0x7f4f1819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f1819ef90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009230 tx=0x7f4f08005e40 comp rx=0 tx=0).stop 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 shutdown_connections 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f181022b0 0x7f4f1819ef90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f4f040a2800 0x7f4f040a4cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181018f0 0x7f4f18197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 --2- 172.21.15.78:0/2154307728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18100700 0x7f4f181979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 >> 172.21.15.78:0/2154307728 conn(0x7f4f180fbc70 msgr2=0x7f4f18068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 shutdown_connections 2024-03-16T11:08:41.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.628+0000 7f4f1edc6640 1 -- 172.21.15.78:0/2154307728 wait complete. 2024-03-16T11:08:41.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T11:08:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.742+0000 7f1d03c6c640 1 -- 172.21.15.78:0/2740850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 msgr2=0x7f1cfc109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/2740850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 0x7f1cfc109910 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009860 tx=0x7f1cf402bb70 comp rx=0 tx=0).stop 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- 172.21.15.78:0/2740850017 shutdown_connections 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/2740850017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 0x7f1cfc109910 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/2740850017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc104b90 0x7f1cfc106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/2740850017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cfc102260 0x7f1cfc104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- 172.21.15.78:0/2740850017 >> 172.21.15.78:0/2740850017 conn(0x7f1cfc0fbc70 msgr2=0x7f1cfc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- 172.21.15.78:0/2740850017 shutdown_connections 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- 172.21.15.78:0/2740850017 wait complete. 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 Processor -- start 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- start start 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc102260 0x7f1cfc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cfc104b90 0x7f1cfc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 0x7f1cfc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc10d330 con 0x7f1cfc102260 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc10d1b0 con 0x7f1cfc1074c0 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d03c6c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc10d4b0 con 0x7f1cfc104b90 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.743+0000 7f1d01a08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc102260 0x7f1cfc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc102260 0x7f1cfc197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52916/0 (socket says 172.21.15.78:52916) 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 -- 172.21.15.78:0/1872366851 learned_addr learned my addr 172.21.15.78:0/1872366851 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d02209640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 0x7f1cfc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01207640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cfc104b90 0x7f1cfc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 -- 172.21.15.78:0/1872366851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cfc104b90 msgr2=0x7f1cfc197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cfc104b90 0x7f1cfc197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 -- 172.21.15.78:0/1872366851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 msgr2=0x7f1cfc19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 0x7f1cfc19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 -- 172.21.15.78:0/1872366851 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cfc19f650 con 0x7f1cfc102260 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01207640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cfc104b90 0x7f1cfc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:08:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d01a08640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc102260 0x7f1cfc197a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800eb90 tx=0x7f1cf800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1cf2ffd640 1 -- 172.21.15.78:0/1872366851 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8004d20 con 0x7f1cfc102260 2024-03-16T11:08:41.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cfc19f940 con 0x7f1cfc102260 2024-03-16T11:08:41.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1cf2ffd640 1 -- 172.21.15.78:0/1872366851 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cf800ee90 con 0x7f1cfc102260 2024-03-16T11:08:41.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1cf2ffd640 1 -- 172.21.15.78:0/1872366851 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8010670 con 0x7f1cfc102260 2024-03-16T11:08:41.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.744+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cfc19fe90 con 0x7f1cfc102260 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.747+0000 7f1cf2ffd640 1 -- 172.21.15.78:0/1872366851 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1cf8005020 con 0x7f1cfc102260 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.747+0000 7f1d0684b740 1 Processor -- start 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.747+0000 7f1d0684b740 1 -- start start 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.747+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e1f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.747+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e24150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.747+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e247c0 0x55afe1e26c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d0684b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55afe1d7f490 con 0x55afe1e1f650 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d0684b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afe1d27c90 con 0x55afe1e1ec60 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d0684b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55afe1d7b530 con 0x55afe1e247c0 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01a08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e24150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d02209640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e1f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01207640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e247c0 0x55afe1e26c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01a08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e24150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52920/0 (socket says 172.21.15.78:52920) 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01a08640 1 -- 172.21.15.78:0/3195827809 learned_addr learned my addr 172.21.15.78:0/3195827809 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1cf2ffd640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1ce80a2800 0x7f1ce80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01207640 1 -- 172.21.15.78:0/3195827809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 msgr2=0x55afe1e1f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01207640 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e1f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01207640 1 -- 172.21.15.78:0/3195827809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 msgr2=0x55afe1e24150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01207640 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e24150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01207640 1 -- 172.21.15.78:0/3195827809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afe1de2500 con 0x55afe1e247c0 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01a08640 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e24150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d02209640 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e1f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d02209640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1ce80a2800 0x7f1ce80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1cf2ffd640 1 -- 172.21.15.78:0/1872366851 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1cf80cf160 con 0x7f1cfc102260 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d01207640 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e247c0 0x55afe1e26c50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc0694a0 tx=0x7f1cec0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78cc89f815bb8dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1d02209640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1ce80a2800 0x7f1ce80a4cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40349e0 tx=0x7f1cf402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.748+0000 7f1ce7fff640 1 -- 172.21.15.78:0/3195827809 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec012450 con 0x55afe1e247c0 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1ce7fff640 1 -- 172.21.15.78:0/3195827809 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cec012aa0 con 0x55afe1e247c0 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1ce7fff640 1 -- 172.21.15.78:0/3195827809 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00f9c0 con 0x55afe1e247c0 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 -- 172.21.15.78:0/3195827809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e247c0 msgr2=0x55afe1e26c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e247c0 0x55afe1e26c50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc0694a0 tx=0x7f1cec0117b0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 -- 172.21.15.78:0/3195827809 shutdown_connections 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e247c0 0x55afe1e26c50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e24150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3195827809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e1f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 -- 172.21.15.78:0/3195827809 >> 172.21.15.78:0/3195827809 conn(0x55afe1e15dc0 msgr2=0x55afe1e161c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 -- 172.21.15.78:0/3195827809 shutdown_connections 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.749+0000 7f1d0684b740 1 -- 172.21.15.78:0/3195827809 wait complete. 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 -- start start 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 Processor -- start 2024-03-16T11:08:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e29100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1f650 0x55afe1e29640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e30680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55afe1e27770 con 0x55afe1e247c0 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afe1d3b230 con 0x55afe1e1f650 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55afe1e278f0 con 0x55afe1e1ec60 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d0684b740 10 client.0 _ll_register_callbacks cb 0x55afe1d77010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d02209640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e30680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d02209640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e30680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52928/0 (socket says 172.21.15.78:52928) 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d02209640 1 -- 172.21.15.78:0/3108727144 learned_addr learned my addr 172.21.15.78:0/3108727144 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d01a08640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e29100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.750+0000 7f1d01207640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1f650 0x55afe1e29640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d02209640 1 -- 172.21.15.78:0/3108727144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 msgr2=0x55afe1e29100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d02209640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e29100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d02209640 1 -- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1f650 msgr2=0x55afe1e29640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d02209640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1f650 0x55afe1e29640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d02209640 1 -- 172.21.15.78:0/3108727144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afe1d4a100 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d01207640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1f650 0x55afe1e29640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d02209640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e30680 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1cf402e010 tx=0x7f1cf40058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47b105ea4f6e3ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1ce57fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1ce57fa640 1 -- 172.21.15.78:0/3108727144 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4050060 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1ce57fa640 1 -- 172.21.15.78:0/3108727144 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1cf402c920 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cec012aa0 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1ce57fa640 1 -- 172.21.15.78:0/3108727144 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1cf4044070 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 10 client.16128 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55afe1a933f0 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1ce57fa640 1 -- 172.21.15.78:0/3108727144 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cec012aa0 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1ce57fa640 1 -- 172.21.15.78:0/3108727144 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1cf4055070 con 0x55afe1e247c0 2024-03-16T11:08:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 10 client.16128 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 10 client.16128 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 20 client.16128 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e34fb0 0x55afe1e37410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 4 client.16128 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.751+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afe1d8c9d0 con 0x55afe1e34fb0 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.752+0000 7f1d01a08640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e34fb0 0x55afe1e37410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.752+0000 7f1ce57fa640 10 client.16128 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.752+0000 7f1d01a08640 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e34fb0 0x55afe1e37410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=953f76979e0a32ed server_cookie=1c77a642f4aab738 in_seq=0 out_seq=0 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1ce57fa640 1 -- 172.21.15.78:0/3108727144 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afe1d8c9d0 con 0x55afe1e34fb0 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1ce57fa640 10 client.16128 handle_command_reply: tid=0 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 client.16128 shutdown 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 msgr2=0x55afe1e30680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e30680 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1cf402e010 tx=0x7f1cf40058b0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 shutdown_connections 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e34fb0 0x55afe1e37410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e30680 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1f650 0x55afe1e29640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3108727144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e29100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1ce57fa640 0 client.16128 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 >> 172.21.15.78:0/3108727144 conn(0x55afe1e15dc0 msgr2=0x55afe1e14fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 shutdown_connections 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 1 -- 172.21.15.78:0/3108727144 wait complete. 2024-03-16T11:08:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.753+0000 7f1d0684b740 20 client.16128 trim_cache size 0 max 16384 2024-03-16T11:08:41.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.766+0000 7f1d0684b740 1 Processor -- start 2024-03-16T11:08:41.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.766+0000 7f1d0684b740 1 -- start start 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.766+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e10e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.766+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e30640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.766+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e247c0 0x55afe1e19730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d0684b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc1a9310 con 0x55afe1e1f650 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d0684b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc1a6be0 con 0x55afe1e247c0 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d0684b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc1a4a10 con 0x55afe1e1ec60 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e30640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e30640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52940/0 (socket says 172.21.15.78:52940) 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 -- 172.21.15.78:0/3607967833 learned_addr learned my addr 172.21.15.78:0/3607967833 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01a08640 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d02209640 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e247c0 0x55afe1e19730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 -- 172.21.15.78:0/3607967833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 msgr2=0x55afe1e10e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e10e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 -- 172.21.15.78:0/3607967833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e247c0 msgr2=0x55afe1e19730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e247c0 0x55afe1e19730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 -- 172.21.15.78:0/3607967833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afe1e11640 con 0x55afe1e1f650 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1d01207640 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e30640 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1cec005fd0 tx=0x7f1cec009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb387a5aa529adc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1ce77fe640 1 -- 172.21.15.78:0/3607967833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec004030 con 0x55afe1e1f650 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1ce77fe640 1 -- 172.21.15.78:0/3607967833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cec0054f0 con 0x55afe1e1f650 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.767+0000 7f1ce77fe640 1 -- 172.21.15.78:0/3607967833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0127e0 con 0x55afe1e1f650 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 -- 172.21.15.78:0/3607967833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 msgr2=0x55afe1e30640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e30640 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1cec005fd0 tx=0x7f1cec009460 comp rx=0 tx=0).stop 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 -- 172.21.15.78:0/3607967833 shutdown_connections 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e247c0 0x55afe1e19730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e1f650 0x55afe1e30640 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 --2- 172.21.15.78:0/3607967833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1ec60 0x55afe1e10e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 -- 172.21.15.78:0/3607967833 >> 172.21.15.78:0/3607967833 conn(0x55afe1e15dc0 msgr2=0x55afe1e19020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 -- 172.21.15.78:0/3607967833 shutdown_connections 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 -- 172.21.15.78:0/3607967833 wait complete. 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 -- start start 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.768+0000 7f1d0684b740 1 Processor -- start 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e2e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1f650 0x55afe1e2e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e2f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc1a50a0 con 0x55afe1e247c0 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc1a6510 con 0x55afe1e1ec60 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc1a5770 con 0x55afe1e1f650 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 10 client.0 _ll_register_callbacks cb 0x7f1cf80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e2f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e2f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52952/0 (socket says 172.21.15.78:52952) 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 -- 172.21.15.78:0/1934581153 learned_addr learned my addr 172.21.15.78:0/1934581153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d01a08640 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e2e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d01207640 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1f650 0x55afe1e2e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 -- 172.21.15.78:0/1934581153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1f650 msgr2=0x55afe1e2e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1f650 0x55afe1e2e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 -- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 msgr2=0x55afe1e2e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e2e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 -- 172.21.15.78:0/1934581153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afe1b5d7d0 con 0x55afe1e247c0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d02209640 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e2f930 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4006010 tx=0x7f1cf402c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6136d113a6d71619 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1ce57fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1ce57fa640 1 -- 172.21.15.78:0/1934581153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf405b040 con 0x55afe1e247c0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1ce57fa640 1 -- 172.21.15.78:0/1934581153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1cf4040b70 con 0x55afe1e247c0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cec0054f0 con 0x55afe1e247c0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1ce57fa640 1 -- 172.21.15.78:0/1934581153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1cf4050040 con 0x55afe1e247c0 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 10 client.16134 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:41.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.769+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55afe1e9b760 con 0x55afe1e247c0 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1ce57fa640 1 -- 172.21.15.78:0/1934581153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cec0054f0 con 0x55afe1e247c0 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1ce57fa640 1 -- 172.21.15.78:0/1934581153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1cf405e070 con 0x55afe1e247c0 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 10 client.16134 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 10 client.16134 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 20 client.16134 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e9c150 0x55afe1e9e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 4 client.16134 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 --> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afe1d86d20 con 0x55afe1e9c150 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d01a08640 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e9c150 0x55afe1e9e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1ce57fa640 10 client.16134 ms_handle_connect on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d01a08640 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e9c150 0x55afe1e9e570 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=82e5b0d45fe3eb17 server_cookie=84e45f66c6d75afc in_seq=0 out_seq=0 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1ce57fa640 1 -- 172.21.15.78:0/1934581153 <== mds.2 v2:172.21.15.98:6836/1752557490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afe1d86d20 con 0x55afe1e9c150 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1ce57fa640 10 client.16134 handle_command_reply: tid=0 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 1 client.16134 shutdown 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 msgr2=0x55afe1e2f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.770+0000 7f1d0684b740 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e2f930 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4006010 tx=0x7f1cf402c950 comp rx=0 tx=0).stop 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 shutdown_connections 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:6836/1752557490,v1:172.21.15.98:6837/1752557490] conn(0x55afe1e9c150 0x55afe1e9e570 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55afe1e247c0 0x55afe1e2f930 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1ce57fa640 0 client.16134 ms_handle_reset on v2:172.21.15.98:6836/1752557490 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afe1e1f650 0x55afe1e2e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 --2- 172.21.15.78:0/1934581153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afe1e1ec60 0x55afe1e2e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 >> 172.21.15.78:0/1934581153 conn(0x55afe1e15dc0 msgr2=0x55afe1e198b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 shutdown_connections 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 1 -- 172.21.15.78:0/1934581153 wait complete. 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d0684b740 20 client.16134 trim_cache size 0 max 16384 2024-03-16T11:08:41.778 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:08:41.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1ce80a2800 msgr2=0x7f1ce80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1ce80a2800 0x7f1ce80a4cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40349e0 tx=0x7f1cf402e040 comp rx=0 tx=0).stop 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc102260 msgr2=0x7f1cfc197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc102260 0x7f1cfc197a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800eb90 tx=0x7f1cf800c6a0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 shutdown_connections 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cfc1074c0 0x7f1cfc19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1ce80a2800 0x7f1ce80a4cc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cfc104b90 0x7f1cfc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 --2- 172.21.15.78:0/1872366851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cfc102260 0x7f1cfc197a30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 >> 172.21.15.78:0/1872366851 conn(0x7f1cfc0fbc70 msgr2=0x7f1cfc102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 shutdown_connections 2024-03-16T11:08:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.771+0000 7f1d03c6c640 1 -- 172.21.15.78:0/1872366851 wait complete. 2024-03-16T11:08:41.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T11:08:41.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 -- 172.21.15.78:0/2972831392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901017f0 msgr2=0x7fb390101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 --2- 172.21.15.78:0/2972831392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901017f0 0x7fb390101c10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb384009900 tx=0x7fb38402bc30 comp rx=0 tx=0).stop 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 -- 172.21.15.78:0/2972831392 shutdown_connections 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 --2- 172.21.15.78:0/2972831392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3901033a0 0x7fb390109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 --2- 172.21.15.78:0/2972831392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3901029e0 0x7fb390102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 --2- 172.21.15.78:0/2972831392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901017f0 0x7fb390101c10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 -- 172.21.15.78:0/2972831392 >> 172.21.15.78:0/2972831392 conn(0x7fb3900fcd80 msgr2=0x7fb3900ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 -- 172.21.15.78:0/2972831392 shutdown_connections 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 -- 172.21.15.78:0/2972831392 wait complete. 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 Processor -- start 2024-03-16T11:08:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 -- start start 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.886+0000 7fb397097640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3901017f0 0x7fb390197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb397097640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3901029e0 0x7fb390197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb397097640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901033a0 0x7fb39019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb397097640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb39010d2f0 con 0x7fb3901033a0 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb397097640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb39010d170 con 0x7fb3901029e0 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb397097640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb39010d470 con 0x7fb3901017f0 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901033a0 0x7fb39019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb394e33640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3901017f0 0x7fb390197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb394e33640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3901017f0 0x7fb390197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:49902/0 (socket says 172.21.15.78:49902) 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901033a0 0x7fb39019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52970/0 (socket says 172.21.15.78:52970) 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3901029e0 0x7fb390197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb394e33640 1 -- 172.21.15.78:0/1061784359 learned_addr learned my addr 172.21.15.78:0/1061784359 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 -- 172.21.15.78:0/1061784359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3901017f0 msgr2=0x7fb390197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3901017f0 0x7fb390197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 -- 172.21.15.78:0/1061784359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3901029e0 msgr2=0x7fb390197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3901029e0 0x7fb390197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 -- 172.21.15.78:0/1061784359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39019f6c0 con 0x7fb3901033a0 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb395634640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901033a0 0x7fb39019f050 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb38800ec50 tx=0x7fb38800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb38dffb640 1 -- 172.21.15.78:0/1061784359 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388003b70 con 0x7fb3901033a0 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb38dffb640 1 -- 172.21.15.78:0/1061784359 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb388003d10 con 0x7fb3901033a0 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39019f9b0 con 0x7fb3901033a0 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.887+0000 7fb38dffb640 1 -- 172.21.15.78:0/1061784359 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388010760 con 0x7fb3901033a0 2024-03-16T11:08:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.888+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39019feb0 con 0x7fb3901033a0 2024-03-16T11:08:41.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.889+0000 7fb38dffb640 1 -- 172.21.15.78:0/1061784359 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb388004020 con 0x7fb3901033a0 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 Processor -- start 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb38dffb640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3800a2800 0x7fb3800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 -- start start 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac2dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac32d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608fac35840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3800a2800 0x7fb3800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5608fab92490 con 0x5608fac2e240 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5608fab3ac90 con 0x5608fac333b0 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb399922740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5608fab8e530 con 0x5608fac2d850 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb38dffb640 1 -- 172.21.15.78:0/1061784359 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb3880cefb0 con 0x7fb3901033a0 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb394e33640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac2dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac32d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb394e33640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac2dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:49914/0 (socket says 172.21.15.78:49914) 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb394e33640 1 -- 172.21.15.78:0/1826467823 learned_addr learned my addr 172.21.15.78:0/1826467823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb395634640 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608fac35840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3800a2800 0x7fb3800a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb390198fb0 tx=0x7fb37c013880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.890+0000 7fb394e33640 1 -- 172.21.15.78:0/1826467823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 msgr2=0x5608fac35840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb394e33640 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608fac35840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb394e33640 1 -- 172.21.15.78:0/1826467823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 msgr2=0x5608fac32d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb394e33640 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac32d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb394e33640 1 -- 172.21.15.78:0/1826467823 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608fabf5500 con 0x5608fac2d850 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb394e33640 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac2dc70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb384036040 tx=0x7fb384009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=342046369542e805 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb37affd640 1 -- 172.21.15.78:0/1826467823 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb384043070 con 0x5608fac2d850 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb37affd640 1 -- 172.21.15.78:0/1826467823 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb384004040 con 0x5608fac2d850 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.891+0000 7fb37affd640 1 -- 172.21.15.78:0/1826467823 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3840055a0 con 0x5608fac2d850 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- 172.21.15.78:0/1826467823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 msgr2=0x5608fac2dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac2dc70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb384036040 tx=0x7fb384009660 comp rx=0 tx=0).stop 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- 172.21.15.78:0/1826467823 shutdown_connections 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608fac35840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac32d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 --2- 172.21.15.78:0/1826467823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac2dc70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- 172.21.15.78:0/1826467823 >> 172.21.15.78:0/1826467823 conn(0x5608fac28dc0 msgr2=0x5608fac291c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- 172.21.15.78:0/1826467823 shutdown_connections 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- 172.21.15.78:0/1826467823 wait complete. 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- start start 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 Processor -- start 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2e240 0x5608fac381f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac333b0 0x5608fac3f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5608fac36300 con 0x5608fac333b0 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5608fab4e230 con 0x5608fac2e240 2024-03-16T11:08:41.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5608fac36480 con 0x5608fac2d850 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.892+0000 7fb399922740 10 client.0 _ll_register_callbacks cb 0x5608fab8a010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb395634640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac333b0 0x5608fac3f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb394e33640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2e240 0x5608fac381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb395634640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac333b0 0x5608fac3f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52994/0 (socket says 172.21.15.78:52994) 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb395634640 1 -- 172.21.15.78:0/688746763 learned_addr learned my addr 172.21.15.78:0/688746763 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb394e33640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:49924/0 (socket says 172.21.15.78:49924) 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb38ffff640 1 -- 172.21.15.78:0/688746763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 msgr2=0x5608fac37cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb38ffff640 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb38ffff640 1 -- 172.21.15.78:0/688746763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac333b0 msgr2=0x5608fac3f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb38ffff640 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac333b0 0x5608fac3f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb38ffff640 1 -- 172.21.15.78:0/688746763 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608fab5d100 con 0x5608fac2e240 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb395634640 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac333b0 0x5608fac3f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb38ffff640 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2e240 0x5608fac381f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x5608fac2e640 tx=0x7fb37c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d551afc6f6930fd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb36ffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb36ffff640 1 -- 172.21.15.78:0/688746763 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c012270 con 0x5608fac2e240 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb36ffff640 1 -- 172.21.15.78:0/688746763 <== mon.2 v2:172.21.15.98:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb37c003ac0 con 0x5608fac2e240 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb384004040 con 0x5608fac2e240 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb36ffff640 1 -- 172.21.15.78:0/688746763 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb37c003cc0 con 0x5608fac2e240 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb399922740 10 client.25976 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5608fa8a63f0 con 0x5608fac2e240 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.893+0000 7fb36ffff640 1 -- 172.21.15.78:0/688746763 <== mon.2 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb384004040 con 0x5608fac2e240 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb36ffff640 1 -- 172.21.15.78:0/688746763 <== mon.2 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb37c01f070 con 0x5608fac2e240 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb399922740 10 client.25976 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb399922740 10 client.25976 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb399922740 20 client.25976 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb399922740 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608fac43b50 0x5608fac45f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb399922740 4 client.25976 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608fab9f9d0 con 0x5608fac43b50 2024-03-16T11:08:41.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb394e33640 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608fac43b50 0x5608fac45f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb36ffff640 10 client.25976 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.894+0000 7fb394e33640 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608fac43b50 0x5608fac45f80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=77b8f850428abbd2 server_cookie=d0488d50554c70e8 in_seq=0 out_seq=0 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb36ffff640 1 -- 172.21.15.78:0/688746763 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5608fab9f9d0 con 0x5608fac43b50 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb36ffff640 10 client.25976 handle_command_reply: tid=0 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 client.25976 shutdown 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2e240 msgr2=0x5608fac381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2e240 0x5608fac381f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x5608fac2e640 tx=0x7fb37c004730 comp rx=0 tx=0).stop 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 shutdown_connections 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608fac43b50 0x5608fac45f80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac333b0 0x5608fac3f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2e240 0x5608fac381f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb36ffff640 0 client.25976 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 --2- 172.21.15.78:0/688746763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 >> 172.21.15.78:0/688746763 conn(0x5608fac28dc0 msgr2=0x5608fac27fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 shutdown_connections 2024-03-16T11:08:41.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 1 -- 172.21.15.78:0/688746763 wait complete. 2024-03-16T11:08:41.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.895+0000 7fb399922740 20 client.25976 trim_cache size 0 max 16384 2024-03-16T11:08:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 Processor -- start 2024-03-16T11:08:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 -- start start 2024-03-16T11:08:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac381f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac333b0 0x5608fac27750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901ab520 con 0x5608fac2e240 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901a8df0 con 0x5608fac2d850 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb399922740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901a6c20 con 0x5608fac333b0 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.908+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac381f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52996/0 (socket says 172.21.15.78:52996) 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 -- 172.21.15.78:0/1021515040 learned_addr learned my addr 172.21.15.78:0/1021515040 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb394e33640 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb395634640 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac333b0 0x5608fac27750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 -- 172.21.15.78:0/1021515040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac333b0 msgr2=0x5608fac27750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac333b0 0x5608fac27750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 -- 172.21.15.78:0/1021515040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2d850 msgr2=0x5608fac37cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 -- 172.21.15.78:0/1021515040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608fa9707d0 con 0x5608fac2e240 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb394e33640 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac381f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb37c010f10 tx=0x7fb37c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60872f4e944a8373 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb37a7fc640 1 -- 172.21.15.78:0/1021515040 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c0039c0 con 0x5608fac2e240 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb37a7fc640 1 -- 172.21.15.78:0/1021515040 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb37c039070 con 0x5608fac2e240 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.909+0000 7fb37a7fc640 1 -- 172.21.15.78:0/1021515040 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c010690 con 0x5608fac2e240 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 -- 172.21.15.78:0/1021515040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 msgr2=0x5608fac381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac381f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb37c010f10 tx=0x7fb37c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 -- 172.21.15.78:0/1021515040 shutdown_connections 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac333b0 0x5608fac27750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608fac381f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 --2- 172.21.15.78:0/1021515040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac2d850 0x5608fac37cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 -- 172.21.15.78:0/1021515040 >> 172.21.15.78:0/1021515040 conn(0x5608fac28dc0 msgr2=0x5608fac2f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 -- 172.21.15.78:0/1021515040 shutdown_connections 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 -- 172.21.15.78:0/1021515040 wait complete. 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 -- start start 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 Processor -- start 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.910+0000 7fb399922740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608faca4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608faca5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608facac120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901a72b0 con 0x5608fac2e240 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901a8720 con 0x5608fac333b0 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901a7980 con 0x5608fac2d850 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 10 client.0 _ll_register_callbacks cb 0x7fb37c004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608faca5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608faca5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53006/0 (socket says 172.21.15.78:53006) 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 -- 172.21.15.78:0/1372624556 learned_addr learned my addr 172.21.15.78:0/1372624556 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb395634640 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608facac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb394e33640 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608faca4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 -- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 msgr2=0x5608faca4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608faca4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 -- 172.21.15.78:0/1372624556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 msgr2=0x5608facac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608facac120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 -- 172.21.15.78:0/1372624556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608facac710 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb38ffff640 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608faca5150 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb37c013d80 tx=0x7fb37c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f0aeee371a075b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb36ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb36ffff640 1 -- 172.21.15.78:0/1372624556 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c020770 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb36ffff640 1 -- 172.21.15.78:0/1372624556 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb37c020910 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb37c039070 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb36ffff640 1 -- 172.21.15.78:0/1372624556 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb37c010430 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 10 client.16152 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.911+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5608facada10 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb36ffff640 1 -- 172.21.15.78:0/1372624556 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb37c039070 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb36ffff640 1 -- 172.21.15.78:0/1372624556 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb37c0297c0 con 0x5608fac2e240 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb399922740 10 client.16152 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb399922740 10 client.16152 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb399922740 20 client.16152 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb399922740 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608facae380 0x5608facb07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb399922740 4 client.16152 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 --> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5608fab99d20 con 0x5608facae380 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb394e33640 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608facae380 0x5608facb07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb36ffff640 10 client.16152 ms_handle_connect on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.912+0000 7fb394e33640 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608facae380 0x5608facb07a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b51cf6e7acbade08 server_cookie=81f50a7375302f16 in_seq=0 out_seq=0 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb36ffff640 1 -- 172.21.15.78:0/1372624556 <== mds.1 v2:172.21.15.153:6838/1504943793 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5608fab99d20 con 0x5608facae380 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb36ffff640 10 client.16152 handle_command_reply: tid=0 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 client.16152 shutdown 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 msgr2=0x5608faca5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608faca5150 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb37c013d80 tx=0x7fb37c009b30 comp rx=0 tx=0).stop 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 shutdown_connections 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:6838/1504943793,v1:172.21.15.153:6839/1504943793] conn(0x5608facae380 0x5608facb07a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5608fac333b0 0x5608facac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb36ffff640 0 client.16152 ms_handle_reset on v2:172.21.15.153:6838/1504943793 2024-03-16T11:08:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5608fac2e240 0x5608faca5150 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 --2- 172.21.15.78:0/1372624556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5608fac2d850 0x5608faca4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 >> 172.21.15.78:0/1372624556 conn(0x5608fac28dc0 msgr2=0x5608fac2f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 shutdown_connections 2024-03-16T11:08:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 1 -- 172.21.15.78:0/1372624556 wait complete. 2024-03-16T11:08:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb399922740 20 client.16152 trim_cache size 0 max 16384 2024-03-16T11:08:41.920 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3800a2800 msgr2=0x7fb3800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3800a2800 0x7fb3800a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb390198fb0 tx=0x7fb37c013880 comp rx=0 tx=0).stop 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901033a0 msgr2=0x7fb39019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901033a0 0x7fb39019f050 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb38800ec50 tx=0x7fb38800c6a0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 shutdown_connections 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3901033a0 0x7fb39019f050 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3901029e0 0x7fb390197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb3800a2800 0x7fb3800a4cc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 --2- 172.21.15.78:0/1061784359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3901017f0 0x7fb390197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.913+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 >> 172.21.15.78:0/1061784359 conn(0x7fb3900fcd80 msgr2=0x7fb390105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.914+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 shutdown_connections 2024-03-16T11:08:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:41.914+0000 7fb397097640 1 -- 172.21.15.78:0/1061784359 wait complete. 2024-03-16T11:08:41.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T11:08:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 -- 172.21.15.78:0/560300394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181022a0 msgr2=0x7f3518108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 --2- 172.21.15.78:0/560300394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181022a0 0x7f3518108b30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3510009ca0 tx=0x7f351002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 -- 172.21.15.78:0/560300394 shutdown_connections 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 --2- 172.21.15.78:0/560300394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181022a0 0x7f3518108b30 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 --2- 172.21.15.78:0/560300394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35181018e0 0x7f3518101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 --2- 172.21.15.78:0/560300394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35181006f0 0x7f3518100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 -- 172.21.15.78:0/560300394 >> 172.21.15.78:0/560300394 conn(0x7f35180fbc80 msgr2=0x7f35180fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 -- 172.21.15.78:0/560300394 shutdown_connections 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 -- 172.21.15.78:0/560300394 wait complete. 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 Processor -- start 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.030+0000 7f351e261640 1 -- start start 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35181006f0 0x7f3518197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181018e0 0x7f3518197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35181022a0 0x7f351819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f351810c220 con 0x7f35181018e0 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f351810c0a0 con 0x7f35181006f0 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f351810c3a0 con 0x7f35181022a0 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181018e0 0x7f3518197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181018e0 0x7f3518197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53024/0 (socket says 172.21.15.78:53024) 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 -- 172.21.15.78:0/3151557014 learned_addr learned my addr 172.21.15.78:0/3151557014 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f35177fe640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35181006f0 0x7f3518197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3517fff640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35181022a0 0x7f351819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 -- 172.21.15.78:0/3151557014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35181022a0 msgr2=0x7f351819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35181022a0 0x7f351819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 -- 172.21.15.78:0/3151557014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35181006f0 msgr2=0x7f3518197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35181006f0 0x7f3518197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 -- 172.21.15.78:0/3151557014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351819f610 con 0x7f35181018e0 2024-03-16T11:08:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3516ffd640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181018e0 0x7f3518197f80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f350800ea60 tx=0x7f350800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3514ff9640 1 -- 172.21.15.78:0/3151557014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350800cde0 con 0x7f35181018e0 2024-03-16T11:08:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351819f900 con 0x7f35181018e0 2024-03-16T11:08:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3514ff9640 1 -- 172.21.15.78:0/3151557014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35080035f0 con 0x7f35181018e0 2024-03-16T11:08:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f3514ff9640 1 -- 172.21.15.78:0/3151557014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3508010690 con 0x7f35181018e0 2024-03-16T11:08:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.031+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351819fe50 con 0x7f35181018e0 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.033+0000 7f3514ff9640 1 -- 172.21.15.78:0/3151557014 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f35080037a0 con 0x7f35181018e0 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 Processor -- start 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 -- start start 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3514ff9640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35000a2800 0x7f35000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781ba080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781bf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555b7811a490 con 0x555b781ba650 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b780c2c90 con 0x555b781bf7c0 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3520e52740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555b78116530 con 0x555b781b9c60 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3517fff640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35000a2800 0x7f35000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3516ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781bf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3516ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781bf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53030/0 (socket says 172.21.15.78:53030) 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3516ffd640 1 -- 172.21.15.78:0/242079949 learned_addr learned my addr 172.21.15.78:0/242079949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3517fff640 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f35177fe640 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781ba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.034+0000 7f3514ff9640 1 -- 172.21.15.78:0/3151557014 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f35080ced60 con 0x7f35181018e0 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f35177fe640 1 -- 172.21.15.78:0/242079949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 msgr2=0x555b781c1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f35177fe640 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781c1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f35177fe640 1 -- 172.21.15.78:0/242079949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 msgr2=0x555b781bf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f35177fe640 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781bf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f35177fe640 1 -- 172.21.15.78:0/242079949 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b7817d500 con 0x555b781b9c60 2024-03-16T11:08:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f3516ffd640 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781bf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f3517fff640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35000a2800 0x7f35000a4cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3510006010 tx=0x7f3510009970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f35177fe640 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781ba080 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f350c009bd0 tx=0x7f350c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9997faf42b5c1145 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f34fdffb640 1 -- 172.21.15.78:0/242079949 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c004220 con 0x555b781b9c60 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f34fdffb640 1 -- 172.21.15.78:0/242079949 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f350c009e40 con 0x555b781b9c60 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.035+0000 7f34fdffb640 1 -- 172.21.15.78:0/242079949 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c018630 con 0x555b781b9c60 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 -- 172.21.15.78:0/242079949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 msgr2=0x555b781ba080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781ba080 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f350c009bd0 tx=0x7f350c011810 comp rx=0 tx=0).stop 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 -- 172.21.15.78:0/242079949 shutdown_connections 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781bf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 --2- 172.21.15.78:0/242079949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781ba080 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 -- 172.21.15.78:0/242079949 >> 172.21.15.78:0/242079949 conn(0x555b781b0dc0 msgr2=0x555b781b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 -- 172.21.15.78:0/242079949 shutdown_connections 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 -- 172.21.15.78:0/242079949 wait complete. 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 -- start start 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 Processor -- start 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.036+0000 7f3520e52740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781bf7c0 0x555b781cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555b781c2770 con 0x555b781bf7c0 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555b780d6230 con 0x555b781ba650 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555b781c28f0 con 0x555b781b9c60 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 10 client.0 _ll_register_callbacks cb 0x555b78112010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:42.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781bf7c0 0x555b781cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f35177fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781bf7c0 0x555b781cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53044/0 (socket says 172.21.15.78:53044) 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 -- 172.21.15.78:0/130254153 learned_addr learned my addr 172.21.15.78:0/130254153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3516ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 -- 172.21.15.78:0/130254153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 msgr2=0x555b781c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 -- 172.21.15.78:0/130254153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781ba650 msgr2=0x555b781c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 -- 172.21.15.78:0/130254153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b780e5100 con 0x555b781bf7c0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3517fff640 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781bf7c0 0x555b781cb690 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3510036000 tx=0x7f351002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21aa7a70f7ef9bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f34f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f34f6ffd640 1 -- 172.21.15.78:0/130254153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510039040 con 0x555b781bf7c0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f34f6ffd640 1 -- 172.21.15.78:0/130254153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f35100047d0 con 0x555b781bf7c0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b781279d0 con 0x555b781bf7c0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f34f6ffd640 1 -- 172.21.15.78:0/130254153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3510034440 con 0x555b781bf7c0 2024-03-16T11:08:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 10 client.16164 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.037+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555b77e2e3f0 con 0x555b781bf7c0 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f34f6ffd640 1 -- 172.21.15.78:0/130254153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555b781279d0 con 0x555b781bf7c0 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f34f6ffd640 1 -- 172.21.15.78:0/130254153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f351003f040 con 0x555b781bf7c0 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f3520e52740 10 client.16164 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f3520e52740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f3520e52740 20 client.16164 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f3520e52740 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b781cffd0 0x555b781d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f3520e52740 4 client.16164 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b78121d20 con 0x555b781cffd0 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f35177fe640 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b781cffd0 0x555b781d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f35177fe640 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b781cffd0 0x555b781d2430 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8de4ba488f734f8c server_cookie=95d9ce7ac5a105cf in_seq=0 out_seq=0 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.038+0000 7f34f6ffd640 10 client.16164 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f34f6ffd640 1 -- 172.21.15.78:0/130254153 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555b78121d20 con 0x555b781cffd0 2024-03-16T11:08:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f34f6ffd640 10 client.16164 handle_command_reply: tid=0 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 client.16164 shutdown 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781bf7c0 msgr2=0x555b781cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781bf7c0 0x555b781cb690 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3510036000 tx=0x7f351002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 shutdown_connections 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b781cffd0 0x555b781d2430 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781bf7c0 0x555b781cb690 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f34f6ffd640 0 client.16164 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 --2- 172.21.15.78:0/130254153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 >> 172.21.15.78:0/130254153 conn(0x555b781b0dc0 msgr2=0x555b781affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 shutdown_connections 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 1 -- 172.21.15.78:0/130254153 wait complete. 2024-03-16T11:08:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.039+0000 7f3520e52740 20 client.16164 trim_cache size 0 max 16384 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 Processor -- start 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 -- start start 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781cb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35181a9290 con 0x555b781b9c60 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35181a6b60 con 0x555b781bf7c0 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3520e52740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35181a4990 con 0x555b781ba650 2024-03-16T11:08:42.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f35177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f35177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53052/0 (socket says 172.21.15.78:53052) 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f35177fe640 1 -- 172.21.15.78:0/3626569312 learned_addr learned my addr 172.21.15.78:0/3626569312 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3517fff640 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.052+0000 7f3516ffd640 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f35177fe640 1 -- 172.21.15.78:0/3626569312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781ba650 msgr2=0x555b781c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f35177fe640 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f35177fe640 1 -- 172.21.15.78:0/3626569312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 msgr2=0x555b781cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f35177fe640 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f35177fe640 1 -- 172.21.15.78:0/3626569312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b77ef87d0 con 0x555b781b9c60 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f35177fe640 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781b9c60 0x555b781c4110 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f350c0058c0 tx=0x7f350c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cd16dbb7fb93eb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f34fd7fa640 1 -- 172.21.15.78:0/3626569312 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c01e040 con 0x555b781b9c60 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f34fd7fa640 1 -- 172.21.15.78:0/3626569312 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f350c02a070 con 0x555b781b9c60 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f34fd7fa640 1 -- 172.21.15.78:0/3626569312 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c005060 con 0x555b781b9c60 2024-03-16T11:08:42.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f3520e52740 1 -- 172.21.15.78:0/3626569312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781b9c60 msgr2=0x555b781c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.053+0000 7f3520e52740 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781b9c60 0x555b781c4110 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f350c0058c0 tx=0x7f350c009da0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- 172.21.15.78:0/3626569312 shutdown_connections 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781bf7c0 0x555b781cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781ba650 0x555b781c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 --2- 172.21.15.78:0/3626569312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781b9c60 0x555b781c4110 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- 172.21.15.78:0/3626569312 >> 172.21.15.78:0/3626569312 conn(0x555b781b0dc0 msgr2=0x555b781b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- 172.21.15.78:0/3626569312 shutdown_connections 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- 172.21.15.78:0/3626569312 wait complete. 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- start start 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 Processor -- start 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781b9c60 0x555b781c8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781c8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781bf7c0 0x555b78234360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35181a5020 con 0x555b781ba650 2024-03-16T11:08:42.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35181a6490 con 0x555b781b9c60 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35181a56f0 con 0x555b781bf7c0 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3520e52740 10 client.0 _ll_register_callbacks cb 0x7f350c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3516ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3516ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53062/0 (socket says 172.21.15.78:53062) 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3516ffd640 1 -- 172.21.15.78:0/358936279 learned_addr learned my addr 172.21.15.78:0/358936279 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f35177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781b9c60 0x555b781c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.054+0000 7f3517fff640 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781bf7c0 0x555b78234360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3516ffd640 1 -- 172.21.15.78:0/358936279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781bf7c0 msgr2=0x555b78234360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3516ffd640 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781bf7c0 0x555b78234360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3516ffd640 1 -- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781b9c60 msgr2=0x555b781c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3516ffd640 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781b9c60 0x555b781c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3516ffd640 1 -- 172.21.15.78:0/358936279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b781cb860 con 0x555b781ba650 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f35177fe640 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781b9c60 0x555b781c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:42.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3516ffd640 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781c8790 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f35080980d0 tx=0x7f350809b190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b273a958c51a42f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f34f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f34f6ffd640 1 -- 172.21.15.78:0/358936279 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350809be00 con 0x555b781ba650 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f34f6ffd640 1 -- 172.21.15.78:0/358936279 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f3508097d70 con 0x555b781ba650 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f350c02a070 con 0x555b781ba650 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f34f6ffd640 1 -- 172.21.15.78:0/358936279 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f350809b700 con 0x555b781ba650 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 10 client.16170 fetch_fsmap learned FSMap version 22 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555b782357f0 con 0x555b781ba650 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f34f6ffd640 1 -- 172.21.15.78:0/358936279 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f350c02a070 con 0x555b781ba650 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f34f6ffd640 1 -- 172.21.15.78:0/358936279 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f350809ab50 con 0x555b781ba650 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 10 client.16170 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 20 client.16170 populate_metadata read hostname 'smithi078' 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b78236270 0x555b78238690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 4 client.16170 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T11:08:42.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.055+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 --> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b7809f6a0 con 0x555b78236270 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f35177fe640 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b78236270 0x555b78238690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f34f6ffd640 10 client.16170 ms_handle_connect on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f35177fe640 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b78236270 0x555b78238690 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c434da5dc312519 server_cookie=1a04e9d9891641b6 in_seq=0 out_seq=0 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f34f6ffd640 1 -- 172.21.15.78:0/358936279 <== mds.0 v2:172.21.15.98:6838/4092843053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b7809f6a0 con 0x555b78236270 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f34f6ffd640 10 client.16170 handle_command_reply: tid=0 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 client.16170 shutdown 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 msgr2=0x555b781c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781c8790 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f35080980d0 tx=0x7f350809b190 comp rx=0 tx=0).stop 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 shutdown_connections 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:6838/4092843053,v1:172.21.15.98:6839/4092843053] conn(0x555b78236270 0x555b78238690 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b781bf7c0 0x555b78234360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b781ba650 0x555b781c8790 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f34f6ffd640 0 client.16170 ms_handle_reset on v2:172.21.15.98:6838/4092843053 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 --2- 172.21.15.78:0/358936279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555b781b9c60 0x555b781c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 >> 172.21.15.78:0/358936279 conn(0x555b781b0dc0 msgr2=0x555b781c4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:42.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 shutdown_connections 2024-03-16T11:08:42.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 1 -- 172.21.15.78:0/358936279 wait complete. 2024-03-16T11:08:42.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.056+0000 7f3520e52740 20 client.16170 trim_cache size 0 max 16384 2024-03-16T11:08:42.064 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35000a2800 msgr2=0x7f35000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35000a2800 0x7f35000a4cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3510006010 tx=0x7f3510009970 comp rx=0 tx=0).stop 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181018e0 msgr2=0x7f3518197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181018e0 0x7f3518197f80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f350800ea60 tx=0x7f350800ef20 comp rx=0 tx=0).stop 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 shutdown_connections 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f35000a2800 0x7f35000a4cc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35181022a0 0x7f351819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35181018e0 0x7f3518197f80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 --2- 172.21.15.78:0/3151557014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35181006f0 0x7f3518197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 >> 172.21.15.78:0/3151557014 conn(0x7f35180fbc80 msgr2=0x7f35180fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:42.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 shutdown_connections 2024-03-16T11:08:42.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:42.057+0000 7f351e261640 1 -- 172.21.15.78:0/3151557014 wait complete. 2024-03-16T11:08:42.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:42 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2154307728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:08:42.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:42 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2154307728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:08:42.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:42 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2154307728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:08:43.069 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-16T11:08:43.069 INFO:tasks.fwd_scrub:done joining 2024-03-16T11:08:43.069 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-16T11:08:43.080 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 -- 172.21.15.78:0/942367565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06141022b0 msgr2=0x7f0614108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 --2- 172.21.15.78:0/942367565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06141022b0 0x7f0614108b40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0604009900 tx=0x7f060402bc30 comp rx=0 tx=0).stop 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 -- 172.21.15.78:0/942367565 shutdown_connections 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 --2- 172.21.15.78:0/942367565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06141022b0 0x7f0614108b40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 --2- 172.21.15.78:0/942367565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 0x7f0614101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 --2- 172.21.15.78:0/942367565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0614100700 0x7f0614100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 -- 172.21.15.78:0/942367565 >> 172.21.15.78:0/942367565 conn(0x7f06140fbc70 msgr2=0x7f06140fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:43.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 -- 172.21.15.78:0/942367565 shutdown_connections 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 -- 172.21.15.78:0/942367565 wait complete. 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 Processor -- start 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 -- start start 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0614100700 0x7f06141979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 0x7f0614197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06141022b0 0x7f061419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.192+0000 7f061b07e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f061410c210 con 0x7f06141018f0 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f061b07e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f061410c090 con 0x7f0614100700 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f061b07e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f061410c390 con 0x7f06141022b0 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 0x7f0614197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 0x7f0614197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53082/0 (socket says 172.21.15.78:53082) 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 -- 172.21.15.78:0/1766023117 learned_addr learned my addr 172.21.15.78:0/1766023117 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:43.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0618e1a640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0614100700 0x7f06141979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f061961b640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06141022b0 0x7f061419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 -- 172.21.15.78:0/1766023117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06141022b0 msgr2=0x7f061419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06141022b0 0x7f061419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 -- 172.21.15.78:0/1766023117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0614100700 msgr2=0x7f06141979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0614100700 0x7f06141979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 -- 172.21.15.78:0/1766023117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061419f5e0 con 0x7f06141018f0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0618e1a640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0614100700 0x7f06141979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0613fff640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 0x7f0614197f00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00ea60 tx=0x7f05fc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0611ffb640 1 -- 172.21.15.78:0/1766023117 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00cde0 con 0x7f06141018f0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061419f8d0 con 0x7f06141018f0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0611ffb640 1 -- 172.21.15.78:0/1766023117 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05fc0035f0 con 0x7f06141018f0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.193+0000 7f0611ffb640 1 -- 172.21.15.78:0/1766023117 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc010690 con 0x7f06141018f0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.194+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061419fe70 con 0x7f06141018f0 2024-03-16T11:08:43.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.195+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06141a0100 con 0x7f06141018f0 2024-03-16T11:08:43.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.195+0000 7f0611ffb640 1 -- 172.21.15.78:0/1766023117 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f05fc0037a0 con 0x7f06141018f0 2024-03-16T11:08:43.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.196+0000 7f0611ffb640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06000a2800 0x7f06000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.196+0000 7f0618e1a640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06000a2800 0x7f06000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.196+0000 7f0611ffb640 1 -- 172.21.15.78:0/1766023117 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f05fc0ced60 con 0x7f06141018f0 2024-03-16T11:08:43.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.196+0000 7f0618e1a640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06000a2800 0x7f06000a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f060c0099a0 tx=0x7f060c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:43.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.200+0000 7f0611ffb640 1 -- 172.21.15.78:0/1766023117 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f05fc09be00 con 0x7f06141018f0 2024-03-16T11:08:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.376+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06141a02d0 con 0x7f06141018f0 2024-03-16T11:08:43.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.377+0000 7f0611ffb640 1 -- 172.21.15.78:0/1766023117 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f05fc09b700 con 0x7f06141018f0 2024-03-16T11:08:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:08:43.378 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06000a2800 msgr2=0x7f06000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06000a2800 0x7f06000a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f060c0099a0 tx=0x7f060c009500 comp rx=0 tx=0).stop 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 msgr2=0x7f0614197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 0x7f0614197f00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00ea60 tx=0x7f05fc00ef20 comp rx=0 tx=0).stop 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 shutdown_connections 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f06000a2800 0x7f06000a4cc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06141022b0 0x7f061419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06141018f0 0x7f0614197f00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 --2- 172.21.15.78:0/1766023117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0614100700 0x7f06141979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 >> 172.21.15.78:0/1766023117 conn(0x7f06140fbc70 msgr2=0x7f0614068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 shutdown_connections 2024-03-16T11:08:43.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.379+0000 7f061b07e640 1 -- 172.21.15.78:0/1766023117 wait complete. 2024-03-16T11:08:43.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:43 smithi078 ceph-mon[32235]: pgmap v430: 97 pgs: 97 active+clean; 22 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.3 MiB/s wr, 1.03k op/s 2024-03-16T11:08:43.391 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:08:43.392 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:08:43.444 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:43 smithi153 ceph-mon[35521]: pgmap v430: 97 pgs: 97 active+clean; 22 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.3 MiB/s wr, 1.03k op/s 2024-03-16T11:08:43.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:43 smithi098 ceph-mon[35848]: pgmap v430: 97 pgs: 97 active+clean; 22 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.3 MiB/s wr, 1.03k op/s 2024-03-16T11:08:43.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 -- 172.21.15.78:0/3968608763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 msgr2=0x7f89a01004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 --2- 172.21.15.78:0/3968608763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a01004d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8998009be0 tx=0x7f899802bef0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 -- 172.21.15.78:0/3968608763 shutdown_connections 2024-03-16T11:08:43.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 --2- 172.21.15.78:0/3968608763 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a01013b0 0x7f89a0101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 --2- 172.21.15.78:0/3968608763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a0100a10 0x7f89a0100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 --2- 172.21.15.78:0/3968608763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a01004d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 -- 172.21.15.78:0/3968608763 >> 172.21.15.78:0/3968608763 conn(0x7f89a00fbc70 msgr2=0x7f89a00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 -- 172.21.15.78:0/3968608763 shutdown_connections 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 -- 172.21.15.78:0/3968608763 wait complete. 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 Processor -- start 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.496+0000 7f89a72b2640 1 -- start start 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a0199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a0100a10 0x7f89a019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a01013b0 0x7f89a01a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010f4f0 con 0x7f89a01013b0 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010f370 con 0x7f89a0100a10 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010f670 con 0x7f89a01000b0 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a0199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a584f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a01013b0 0x7f89a01a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a0199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:50022/0 (socket says 172.21.15.78:50022) 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 -- 172.21.15.78:0/2073145054 learned_addr learned my addr 172.21.15.78:0/2073145054 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a484d640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a0100a10 0x7f89a019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 -- 172.21.15.78:0/2073145054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a0100a10 msgr2=0x7f89a019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a0100a10 0x7f89a019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 -- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a01013b0 msgr2=0x7f89a01a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a01013b0 0x7f89a01a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 -- 172.21.15.78:0/2073145054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a01a1860 con 0x7f89a01000b0 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a584f640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a01013b0 0x7f89a01a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a504e640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a0199c50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8998004af0 tx=0x7f8998033c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89967fc640 1 -- 172.21.15.78:0/2073145054 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998043020 con 0x7f89a01000b0 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89967fc640 1 -- 172.21.15.78:0/2073145054 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f899802ca80 con 0x7f89a01000b0 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a01a1af0 con 0x7f89a01000b0 2024-03-16T11:08:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89967fc640 1 -- 172.21.15.78:0/2073145054 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998034430 con 0x7f89a01000b0 2024-03-16T11:08:43.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.497+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a01a2040 con 0x7f89a01000b0 2024-03-16T11:08:43.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.498+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a010b0e0 con 0x7f89a01000b0 2024-03-16T11:08:43.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.499+0000 7f89967fc640 1 -- 172.21.15.78:0/2073145054 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f8998046020 con 0x7f89a01000b0 2024-03-16T11:08:43.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.500+0000 7f89967fc640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f898c0a2800 0x7f898c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.500+0000 7f89a484d640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f898c0a2800 0x7f898c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.500+0000 7f89967fc640 1 -- 172.21.15.78:0/2073145054 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f89980f1180 con 0x7f89a01000b0 2024-03-16T11:08:43.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.500+0000 7f89a484d640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f898c0a2800 0x7f898c0a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f89900095f0 tx=0x7f8990005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:43.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.505+0000 7f89967fc640 1 -- 172.21.15.78:0/2073145054 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f899804d020 con 0x7f89a01000b0 2024-03-16T11:08:43.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.681+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f89a0062480 con 0x7f89a01000b0 2024-03-16T11:08:43.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.682+0000 7f89967fc640 1 -- 172.21.15.78:0/2073145054 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f89980be220 con 0x7f89a01000b0 2024-03-16T11:08:43.682 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:08:43.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f898c0a2800 msgr2=0x7f898c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f898c0a2800 0x7f898c0a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f89900095f0 tx=0x7f8990005f90 comp rx=0 tx=0).stop 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 msgr2=0x7f89a0199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a0199c50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8998004af0 tx=0x7f8998033c90 comp rx=0 tx=0).stop 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 shutdown_connections 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a01013b0 0x7f89a01a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a0100a10 0x7f89a019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f898c0a2800 0x7f898c0a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 --2- 172.21.15.78:0/2073145054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a01000b0 0x7f89a0199c50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 >> 172.21.15.78:0/2073145054 conn(0x7f89a00fbc70 msgr2=0x7f89a00fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 shutdown_connections 2024-03-16T11:08:43.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.684+0000 7f89a72b2640 1 -- 172.21.15.78:0/2073145054 wait complete. 2024-03-16T11:08:43.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:08:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 -- 172.21.15.78:0/4086344981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a981006e0 msgr2=0x7f2a98100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4086344981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a981006e0 0x7f2a98100b00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009900 tx=0x7f2a8802bc30 comp rx=0 tx=0).stop 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 -- 172.21.15.78:0/4086344981 shutdown_connections 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4086344981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a98102290 0x7f2a98108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4086344981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 0x7f2a98101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4086344981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a981006e0 0x7f2a98100b00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 -- 172.21.15.78:0/4086344981 >> 172.21.15.78:0/4086344981 conn(0x7f2a980fbc70 msgr2=0x7f2a980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 -- 172.21.15.78:0/4086344981 shutdown_connections 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 -- 172.21.15.78:0/4086344981 wait complete. 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 Processor -- start 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 -- start start 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a981006e0 0x7f2a98197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 0x7f2a98197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.800+0000 7f2a9e861640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a98102290 0x7f2a9819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9e861640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9810d330 con 0x7f2a981018d0 2024-03-16T11:08:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9e861640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9810d1b0 con 0x7f2a98102290 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9e861640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9810d4b0 con 0x7f2a981006e0 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 0x7f2a98197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a98102290 0x7f2a9819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 0x7f2a98197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53102/0 (socket says 172.21.15.78:53102) 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a97fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a981006e0 0x7f2a98197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a977fe640 1 -- 172.21.15.78:0/4179305211 learned_addr learned my addr 172.21.15.78:0/4179305211 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a98102290 0x7f2a9819f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:39228/0 (socket says 172.21.15.78:39228) 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 -- 172.21.15.78:0/4179305211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a981006e0 msgr2=0x7f2a98197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a981006e0 0x7f2a98197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 -- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 msgr2=0x7f2a98197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 0x7f2a98197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 -- 172.21.15.78:0/4179305211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9819f690 con 0x7f2a98102290 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a977fe640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 0x7f2a98197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9cdfe640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a98102290 0x7f2a9819f020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009fd0 tx=0x7f2a9000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:43.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a957fa640 1 -- 172.21.15.78:0/4179305211 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a9000ce20 con 0x7f2a98102290 2024-03-16T11:08:43.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a957fa640 1 -- 172.21.15.78:0/4179305211 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a90005020 con 0x7f2a98102290 2024-03-16T11:08:43.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9819f980 con 0x7f2a98102290 2024-03-16T11:08:43.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a957fa640 1 -- 172.21.15.78:0/4179305211 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90010690 con 0x7f2a98102290 2024-03-16T11:08:43.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.801+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9819fed0 con 0x7f2a98102290 2024-03-16T11:08:43.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.802+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a98107b40 con 0x7f2a98102290 2024-03-16T11:08:43.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.803+0000 7f2a957fa640 1 -- 172.21.15.78:0/4179305211 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2a900051d0 con 0x7f2a98102290 2024-03-16T11:08:43.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.804+0000 7f2a957fa640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2a800a2800 0x7f2a800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:43.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.804+0000 7f2a97fff640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2a800a2800 0x7f2a800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:43.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.804+0000 7f2a957fa640 1 -- 172.21.15.78:0/4179305211 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f2a900cf130 con 0x7f2a98102290 2024-03-16T11:08:43.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.804+0000 7f2a97fff640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2a800a2800 0x7f2a800a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2a88005ab0 tx=0x7f2a88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:43.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.809+0000 7f2a957fa640 1 -- 172.21.15.78:0/4179305211 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a9009c1d0 con 0x7f2a98102290 2024-03-16T11:08:43.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.981+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2a980623e0 con 0x7f2a98102290 2024-03-16T11:08:43.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.981+0000 7f2a957fa640 1 -- 172.21.15.78:0/4179305211 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f2a9009bad0 con 0x7f2a98102290 2024-03-16T11:08:43.982 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2a800a2800 msgr2=0x7f2a800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2a800a2800 0x7f2a800a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2a88005ab0 tx=0x7f2a88005fd0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a98102290 msgr2=0x7f2a9819f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a98102290 0x7f2a9819f020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009fd0 tx=0x7f2a9000ef10 comp rx=0 tx=0).stop 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 shutdown_connections 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a98102290 0x7f2a9819f020 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a981018d0 0x7f2a98197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f2a800a2800 0x7f2a800a4cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 --2- 172.21.15.78:0/4179305211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a981006e0 0x7f2a98197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 >> 172.21.15.78:0/4179305211 conn(0x7f2a980fbc70 msgr2=0x7f2a98104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 shutdown_connections 2024-03-16T11:08:43.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:43.983+0000 7f2a9e861640 1 -- 172.21.15.78:0/4179305211 wait complete. 2024-03-16T11:08:43.995 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:08:44.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 -- 172.21.15.78:0/1579414208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff934102290 msgr2=0x7ff934108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 --2- 172.21.15.78:0/1579414208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff934102290 0x7ff934108b20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009ca0 tx=0x7ff92c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 -- 172.21.15.78:0/1579414208 shutdown_connections 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 --2- 172.21.15.78:0/1579414208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff934102290 0x7ff934108b20 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 --2- 172.21.15.78:0/1579414208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9341018d0 0x7ff934101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 --2- 172.21.15.78:0/1579414208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9341006e0 0x7ff934100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 -- 172.21.15.78:0/1579414208 >> 172.21.15.78:0/1579414208 conn(0x7ff9340fbc70 msgr2=0x7ff9340fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 -- 172.21.15.78:0/1579414208 shutdown_connections 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 -- 172.21.15.78:0/1579414208 wait complete. 2024-03-16T11:08:44.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 Processor -- start 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.099+0000 7ff93c468640 1 -- start start 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93c468640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9341006e0 0x7ff934197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93c468640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9341018d0 0x7ff934197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93c468640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff934102290 0x7ff93419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93c468640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff93410d330 con 0x7ff9341006e0 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93c468640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff93410d1b0 con 0x7ff934102290 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93c468640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff93410d4b0 con 0x7ff9341018d0 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9341006e0 0x7ff934197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9341006e0 0x7ff934197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53112/0 (socket says 172.21.15.78:53112) 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 -- 172.21.15.78:0/2059866338 learned_addr learned my addr 172.21.15.78:0/2059866338 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff939a03640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9341018d0 0x7ff934197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93aa05640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff934102290 0x7ff93419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 -- 172.21.15.78:0/2059866338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9341018d0 msgr2=0x7ff934197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9341018d0 0x7ff934197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 -- 172.21.15.78:0/2059866338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff934102290 msgr2=0x7ff93419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff934102290 0x7ff93419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 -- 172.21.15.78:0/2059866338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93419f630 con 0x7ff9341006e0 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff939a03640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9341018d0 0x7ff934197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93aa05640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff934102290 0x7ff93419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93a204640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9341006e0 0x7ff934197a10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e9c0 tx=0x7ff93000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff92b7fe640 1 -- 172.21.15.78:0/2059866338 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93000cd80 con 0x7ff9341006e0 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff92b7fe640 1 -- 172.21.15.78:0/2059866338 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9300035f0 con 0x7ff9341006e0 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93419f920 con 0x7ff9341006e0 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.100+0000 7ff92b7fe640 1 -- 172.21.15.78:0/2059866338 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930010690 con 0x7ff9341006e0 2024-03-16T11:08:44.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.101+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff93419fe70 con 0x7ff9341006e0 2024-03-16T11:08:44.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.102+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff934107a70 con 0x7ff9341006e0 2024-03-16T11:08:44.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.102+0000 7ff92b7fe640 1 -- 172.21.15.78:0/2059866338 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ff9300037a0 con 0x7ff9341006e0 2024-03-16T11:08:44.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.103+0000 7ff92b7fe640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9200a2880 0x7ff9200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.103+0000 7ff939a03640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9200a2880 0x7ff9200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.103+0000 7ff92b7fe640 1 -- 172.21.15.78:0/2059866338 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff9300cef00 con 0x7ff9341006e0 2024-03-16T11:08:44.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.103+0000 7ff939a03640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9200a2880 0x7ff9200a4d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff934198f20 tx=0x7ff924005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:44.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.108+0000 7ff92b7fe640 1 -- 172.21.15.78:0/2059866338 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff93009bfa0 con 0x7ff9341006e0 2024-03-16T11:08:44.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.286+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff93402d1b0 con 0x7ff9341006e0 2024-03-16T11:08:44.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.286+0000 7ff92b7fe640 1 -- 172.21.15.78:0/2059866338 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7ff93009b8a0 con 0x7ff9341006e0 2024-03-16T11:08:44.287 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9200a2880 msgr2=0x7ff9200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9200a2880 0x7ff9200a4d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff934198f20 tx=0x7ff924005f90 comp rx=0 tx=0).stop 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9341006e0 msgr2=0x7ff934197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9341006e0 0x7ff934197a10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e9c0 tx=0x7ff93000ee80 comp rx=0 tx=0).stop 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 shutdown_connections 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff934102290 0x7ff93419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff9200a2880 0x7ff9200a4d40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9341018d0 0x7ff934197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 --2- 172.21.15.78:0/2059866338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9341006e0 0x7ff934197a10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 >> 172.21.15.78:0/2059866338 conn(0x7ff9340fbc70 msgr2=0x7ff934104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 shutdown_connections 2024-03-16T11:08:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.288+0000 7ff93c468640 1 -- 172.21.15.78:0/2059866338 wait complete. 2024-03-16T11:08:44.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1766023117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:08:44.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2073145054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:08:44.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:44 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4179305211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:08:44.300 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-16T11:08:44.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 -- 172.21.15.78:0/1809016376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82101022b0 msgr2=0x7f8210108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 --2- 172.21.15.78:0/1809016376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82101022b0 0x7f8210108b40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f820c009ca0 tx=0x7f820c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 -- 172.21.15.78:0/1809016376 shutdown_connections 2024-03-16T11:08:44.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 --2- 172.21.15.78:0/1809016376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82101022b0 0x7f8210108b40 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 --2- 172.21.15.78:0/1809016376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82101018f0 0x7f8210101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 --2- 172.21.15.78:0/1809016376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8210100700 0x7f8210100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 -- 172.21.15.78:0/1809016376 >> 172.21.15.78:0/1809016376 conn(0x7f82100fbc90 msgr2=0x7f82100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 -- 172.21.15.78:0/1809016376 shutdown_connections 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 -- 172.21.15.78:0/1809016376 wait complete. 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 Processor -- start 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.400+0000 7f8217bcd640 1 -- start start 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8210100700 0x7f821006aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82101018f0 0x7f821019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82101022b0 0x7f821019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f821010c230 con 0x7f8210100700 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f821010c0b0 con 0x7f82101018f0 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f821010c3b0 con 0x7f82101022b0 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8210100700 0x7f821006aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8210100700 0x7f821006aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53124/0 (socket says 172.21.15.78:53124) 2024-03-16T11:08:44.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 -- 172.21.15.78:0/139794450 learned_addr learned my addr 172.21.15.78:0/139794450 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f821616a640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82101022b0 0x7f821019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215168640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82101018f0 0x7f821019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 -- 172.21.15.78:0/139794450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82101022b0 msgr2=0x7f821019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82101022b0 0x7f821019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 -- 172.21.15.78:0/139794450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82101018f0 msgr2=0x7f821019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82101018f0 0x7f821019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 -- 172.21.15.78:0/139794450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821019f610 con 0x7f8210100700 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8215969640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8210100700 0x7f821006aa00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f820800e9c0 tx=0x7f820800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8206ffd640 1 -- 172.21.15.78:0/139794450 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820800cd80 con 0x7f8210100700 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821019f900 con 0x7f8210100700 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8206ffd640 1 -- 172.21.15.78:0/139794450 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82080035f0 con 0x7f8210100700 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8206ffd640 1 -- 172.21.15.78:0/139794450 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208010690 con 0x7f8210100700 2024-03-16T11:08:44.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.401+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821019fe00 con 0x7f8210100700 2024-03-16T11:08:44.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.402+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8210068a20 con 0x7f8210100700 2024-03-16T11:08:44.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.403+0000 7f8206ffd640 1 -- 172.21.15.78:0/139794450 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f82080037a0 con 0x7f8210100700 2024-03-16T11:08:44.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.404+0000 7f8206ffd640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81fc0a2800 0x7f81fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:08:44.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.404+0000 7f8215168640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81fc0a2800 0x7f81fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:08:44.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.404+0000 7f8206ffd640 1 -- 172.21.15.78:0/139794450 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f82080cef40 con 0x7f8210100700 2024-03-16T11:08:44.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.404+0000 7f8215168640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81fc0a2800 0x7f81fc0a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f821006a770 tx=0x7f8200005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:08:44.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.409+0000 7f8206ffd640 1 -- 172.21.15.78:0/139794450 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f820809bfe0 con 0x7f8210100700 2024-03-16T11:08:44.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1766023117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:08:44.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2073145054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:08:44.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:44 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4179305211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:08:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1766023117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:08:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2073145054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:08:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:44 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4179305211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:08:44.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.630+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f82100623e0 con 0x7f8210100700 2024-03-16T11:08:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.638+0000 7f8206ffd640 1 -- 172.21.15.78:0/139794450 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+739831 (secure 0 0 0) 0x7f820809b8e0 con 0x7f8210100700 2024-03-16T11:08:44.650 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:08:44.650 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:44.650 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 20, 2024-03-16T11:08:44.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "active_gid": 14152, 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "active_name": "x", 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addrs": { 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4254906951 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4254906951 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addr": "172.21.15.78:6801/4254906951", 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "active_change": "2024-03-16T10:57:01.884129+0000", 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "active_mgr_features": 4540701547738038271, 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "available": true, 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "standbys": [ 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 24107, 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "y", 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540701547738038271, 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-16T11:08:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-16T11:08:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T11:08:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-16T11:08:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-16T11:08:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-16T11:08:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T11:08:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T11:08:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T11:08:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T11:08:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-03-16T11:08:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-16T11:08:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-16T11:08:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-16T11:08:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-16T11:08:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-16T11:08:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T11:08:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T11:08:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T11:08:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T11:08:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T11:08:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T11:08:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-16T11:08:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T11:08:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T11:08:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-16T11:08:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T11:08:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T11:08:44.751 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.753 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.753 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T11:08:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T11:08:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T11:08:44.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T11:08:44.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "{}", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T11:08:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T11:08:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T11:08:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-16T11:08:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-16T11:08:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T11:08:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T11:08:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-16T11:08:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-16T11:08:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-16T11:08:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-16T11:08:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-16T11:08:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.980 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-16T11:08:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-16T11:08:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-16T11:08:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-16T11:08:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-16T11:08:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-16T11:08:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-16T11:08:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-16T11:08:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-16T11:08:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 24109, 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540701547738038271, 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T11:08:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-16T11:08:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-16T11:08:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-03-16T11:08:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T11:08:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-16T11:08:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T11:08:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T11:08:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-03-16T11:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-16T11:08:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-16T11:08:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-16T11:08:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-16T11:08:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T11:08:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-16T11:08:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T11:08:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T11:08:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T11:08:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T11:08:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T11:08:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T11:08:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T11:08:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T11:08:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T11:08:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T11:08:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T11:08:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "{}", 2024-03-16T11:08:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T11:08:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T11:08:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T11:08:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T11:08:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-16T11:08:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T11:08:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-16T11:08:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T11:08:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T11:08:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T11:08:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-16T11:08:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-16T11:08:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-16T11:08:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-16T11:08:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-16T11:08:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-16T11:08:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-16T11:08:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-16T11:08:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-16T11:08:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T11:08:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-16T11:08:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-16T11:08:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-16T11:08:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-16T11:08:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-16T11:08:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-16T11:08:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-16T11:08:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T11:08:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-16T11:08:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-16T11:08:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-16T11:08:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-16T11:08:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-16T11:08:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-16T11:08:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-16T11:08:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-16T11:08:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-16T11:08:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-16T11:08:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T11:08:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "modules": [ 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard", 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "iostat", 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "nfs", 2024-03-16T11:08:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "restful" 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-16T11:08:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-16T11:08:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-16T11:08:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-16T11:08:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-16T11:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T11:08:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T11:08:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-16T11:08:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-16T11:08:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-16T11:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-16T11:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-16T11:08:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-16T11:08:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T11:08:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-16T11:08:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-16T11:08:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-16T11:08:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-16T11:08:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-16T11:08:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-16T11:08:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-16T11:08:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-16T11:08:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T11:08:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-16T11:08:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-16T11:08:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T11:08:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-16T11:08:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-16T11:08:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T11:08:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T11:08:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T11:08:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T11:08:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T11:08:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T11:08:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T11:08:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-16T11:08:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T11:08:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-16T11:08:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-16T11:08:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-16T11:08:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-16T11:08:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-16T11:08:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T11:08:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-16T11:08:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-16T11:08:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-16T11:08:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-16T11:08:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-16T11:08:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-16T11:08:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T11:08:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T11:08:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T11:08:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T11:08:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "{}", 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T11:08:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-16T11:08:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-16T11:08:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-16T11:08:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-16T11:08:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-16T11:08:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-16T11:08:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-16T11:08:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-16T11:08:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-16T11:08:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-16T11:08:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-16T11:08:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-16T11:08:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-16T11:08:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-16T11:08:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-16T11:08:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-16T11:08:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-16T11:08:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-16T11:08:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-16T11:08:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-16T11:08:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-16T11:08:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-16T11:08:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T11:08:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-16T11:08:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T11:08:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-16T11:08:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-16T11:08:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-16T11:08:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-16T11:08:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-16T11:08:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-16T11:08:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-16T11:08:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T11:08:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-16T11:08:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-16T11:08:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-16T11:08:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-16T11:08:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-16T11:08:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-16T11:08:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-16T11:08:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-16T11:08:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-16T11:08:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-16T11:08:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-16T11:08:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-16T11:08:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-16T11:08:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-16T11:08:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-16T11:08:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-16T11:08:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-16T11:08:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T11:08:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-16T11:08:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-03-16T11:08:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-16T11:08:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-16T11:08:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-16T11:08:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-16T11:08:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "services": { 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard": "https://172.21.15.78:8443/" 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "always_on_modules": { 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus": [ 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-16T11:08:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific": [ 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-16T11:08:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy": [ 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "reef": [ 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-16T11:08:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "squid": [ 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-16T11:08:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "last_failure_osd_epoch": 3, 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "active_clients": [ 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-16T11:08:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2941755181 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "libcephsqlite", 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3714764002 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2346392144 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-16T11:08:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-16T11:08:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-16T11:08:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3417580197 2024-03-16T11:08:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-16T11:08:46.064 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:46.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:45 smithi153 ceph-mon[35521]: pgmap v431: 97 pgs: 97 active+clean; 1.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 MiB/s wr, 299 op/s 2024-03-16T11:08:46.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:45 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2059866338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:08:46.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:45 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/139794450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T11:08:46.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:45 smithi098 ceph-mon[35848]: pgmap v431: 97 pgs: 97 active+clean; 1.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 MiB/s wr, 299 op/s 2024-03-16T11:08:46.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:45 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2059866338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:08:46.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:45 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/139794450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T11:08:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81fc0a2800 msgr2=0x7f81fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81fc0a2800 0x7f81fc0a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f821006a770 tx=0x7f8200005f90 comp rx=0 tx=0).stop 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8210100700 msgr2=0x7f821006aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8210100700 0x7f821006aa00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f820800e9c0 tx=0x7f820800ee80 comp rx=0 tx=0).stop 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 shutdown_connections 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f81fc0a2800 0x7f81fc0a4cc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82101022b0 0x7f821019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82101018f0 0x7f821019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 --2- 172.21.15.78:0/139794450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8210100700 0x7f821006aa00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 >> 172.21.15.78:0/139794450 conn(0x7f82100fbc90 msgr2=0x7f8210104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 shutdown_connections 2024-03-16T11:08:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:08:44.644+0000 7f8217bcd640 1 -- 172.21.15.78:0/139794450 wait complete. 2024-03-16T11:08:46.073 DEBUG:tasks.check_counter:Getting stats from a 2024-03-16T11:08:46.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-16T11:08:46.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:45 smithi078 ceph-mon[32235]: pgmap v431: 97 pgs: 97 active+clean; 1.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 MiB/s wr, 299 op/s 2024-03-16T11:08:46.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:45 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2059866338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:08:46.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:45 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/139794450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T11:08:46.327 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:08:47.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:47 smithi078 ceph-mon[32235]: pgmap v432: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 KiB/s wr, 62 op/s 2024-03-16T11:08:47.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:47 smithi098 ceph-mon[35848]: pgmap v432: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 KiB/s wr, 62 op/s 2024-03-16T11:08:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:47 smithi153 ceph-mon[35521]: pgmap v432: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 KiB/s wr, 62 op/s 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.001207146, 2024-03-16T11:08:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000269251, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 92, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 85, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 716224, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 50080, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.017017299, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.004685057, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.015287321, 2024-03-16T11:08:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.001481862, 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 85, 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.004786318, 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000056309 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 716224, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 50080 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 1, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 62, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 160, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 88576, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 3, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 2, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.005360752, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.003475129, 2024-03-16T11:08:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001322447, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 60, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.002822271, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000047037 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 88576 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:08:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:08:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:08:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148903, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 29, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-16T11:08:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-16T11:08:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 89, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 700867, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 89, 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 89, 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 700867, 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:47.996 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:48.646 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-16T11:08:48.646 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-16T11:08:48.646 DEBUG:tasks.check_counter:Getting stats from d 2024-03-16T11:08:48.647 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-16T11:08:48.890 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:08:49.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:49 smithi078 ceph-mon[32235]: pgmap v433: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-16T11:08:49.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:49 smithi098 ceph-mon[35848]: pgmap v433: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-16T11:08:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:49 smithi153 ceph-mon[35521]: pgmap v433: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-16T11:08:50.805 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:50.805 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:08:50.805 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 94, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 89, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 714864, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 53472, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.018971995, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.004917426, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.016314865, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.001678323, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 88, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.003666184, 2024-03-16T11:08:50.806 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000041661 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 714864, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 53472 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 5, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 69, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 3152, 2024-03-16T11:08:50.807 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 89904, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 3, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 2, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.006093632, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.003930703, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001473257, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 68, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.003745514, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000055081 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:50.808 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 89904 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.000335781, 2024-03-16T11:08:50.809 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000283487, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:08:50.810 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-16T11:08:50.811 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:08:50.813 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:08:50.814 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148863, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 28, 2024-03-16T11:08:50.815 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-03-16T11:08:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 95, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 701647, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 95, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-16T11:08:50.817 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 95, 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 701647, 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:50.818 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:51.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:51 smithi098 ceph-mon[35848]: pgmap v434: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:08:51.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:51 smithi078 ceph-mon[32235]: pgmap v434: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:08:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:51 smithi153 ceph-mon[35521]: pgmap v434: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:08:51.576 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-16T11:08:51.576 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-16T11:08:51.576 DEBUG:tasks.check_counter:Getting stats from g 2024-03-16T11:08:51.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-16T11:08:51.831 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:08:53.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:53 smithi078 ceph-mon[32235]: pgmap v435: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:08:53.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:53 smithi098 ceph-mon[35848]: pgmap v435: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:08:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:53 smithi153 ceph-mon[35521]: pgmap v435: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-16T11:08:53.569 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:08:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-03-16T11:08:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-03-16T11:08:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-03-16T11:08:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-03-16T11:08:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-03-16T11:08:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-03-16T11:08:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.001283738, 2024-03-16T11:08:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T11:08:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000264450, 2024-03-16T11:08:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:08:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:08:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:08:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 95, 2024-03-16T11:08:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 88, 2024-03-16T11:08:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 717040, 2024-03-16T11:08:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 52000, 2024-03-16T11:08:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.016790611, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.004984354, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.015020882, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.001563128, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 88, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.004451939, 2024-03-16T11:08:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000050590 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 717040, 2024-03-16T11:08:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 52000 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 1, 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 64, 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 160, 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 88928, 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 3, 2024-03-16T11:08:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 2, 2024-03-16T11:08:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.005216873, 2024-03-16T11:08:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.003272387, 2024-03-16T11:08:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001313846, 2024-03-16T11:08:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 62, 2024-03-16T11:08:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.002558298, 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000041262 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 88928 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-16T11:08:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:08:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:08:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:08:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:08:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:08:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-16T11:08:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:08:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:08:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:08:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148903, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 29, 2024-03-16T11:08:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-16T11:08:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 92, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 701257, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 92, 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 92, 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 701257, 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:53.587 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:54.252 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-16T11:08:54.252 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-16T11:08:54.252 DEBUG:tasks.check_counter:Getting stats from j 2024-03-16T11:08:54.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-16T11:08:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:08:55.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:55 smithi078 ceph-mon[32235]: pgmap v436: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T11:08:55.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:55 smithi098 ceph-mon[35848]: pgmap v436: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T11:08:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:55 smithi153 ceph-mon[35521]: pgmap v436: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T11:08:56.120 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-16T11:08:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:08:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-03-16T11:08:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-03-16T11:08:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-03-16T11:08:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-03-16T11:08:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.001078470, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000267424, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 5, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 71, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 3152, 2024-03-16T11:08:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 90256, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 3, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 2, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.006014459, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.003774064, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001612648, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 70, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.003807005, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000054385 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 90256 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 96, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 91, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 715408, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 54752, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-03-16T11:08:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.018354354, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.005234543, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.016473461, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.001660194, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 90, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.003853892, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000042821 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 715408, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 54752 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-16T11:08:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:08:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:08:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 149871, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 29, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:08:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-16T11:08:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:08:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-16T11:08:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-03-16T11:08:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-03-16T11:08:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:08:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:08:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-16T11:08:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 97, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 701907, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 97, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 97, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 701907, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-16T11:08:56.133 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-16T11:08:56.835 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-16T11:08:56.835 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-16T11:08:56.835 DEBUG:tasks.check_counter:Getting stats from b 2024-03-16T11:08:56.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-16T11:08:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:08:57.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:57 smithi098 ceph-mon[35848]: pgmap v437: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-16T11:08:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:57 smithi153 ceph-mon[35521]: pgmap v437: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-16T11:08:57.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:57 smithi078 ceph-mon[32235]: pgmap v437: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 921, 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 617, 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1029175, 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 758196, 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 55, 2024-03-16T11:08:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.085163285, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.017592778, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.049513840, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.020750516, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 616, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.273208081, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000443519 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 752, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000107582, 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000143 2024-03-16T11:08:58.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 843376, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 94192 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 464, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 468, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 166577, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 564676, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 28, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 6, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.036267944, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011099540, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.015335816, 2024-03-16T11:08:58.836 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.008557626, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 467, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1.030970858, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002207646 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 337, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000060871, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000180 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 195200 2024-03-16T11:08:58.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 689, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 639, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 208263, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 864064, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 10, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 6, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.065028187, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.030921719, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.017518354, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.014857797, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 637, 2024-03-16T11:08:58.838 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1.521593979, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002388687 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 353, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000073129, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000207 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-03-16T11:08:58.839 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 462, 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.139508577, 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000301966 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 83, 2024-03-16T11:08:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.002363907, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000028480 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 51, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 51, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 51, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.067798785, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001329387 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-03-16T11:08:58.841 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 55, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 24, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 4, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 20, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 17, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 2, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 2, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 2, 2024-03-16T11:08:58.842 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 75, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 66, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 70, 2024-03-16T11:08:58.843 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 4, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 234, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 70, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 42, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 5, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 37, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 28, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 20, 2024-03-16T11:08:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 30, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 98, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 75, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 2, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 33 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 4, 2024-03-16T11:08:58.845 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 1, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 6, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 5, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-03-16T11:08:58.846 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 5, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 155, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 132, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 132, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 23, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 14, 2024-03-16T11:08:58.847 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 13, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 12, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 12, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 1, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 1, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 1, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4729057, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 4803492, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 0, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 126, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 4.549925330, 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.036110518 2024-03-16T11:08:58.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 0 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 24, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 34, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 10, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 19, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 20, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 1, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 24, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 34, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 10, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 2, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 20, 2024-03-16T11:08:58.849 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 18, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 54728, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 231804 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 75, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 51, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 15, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 8, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.008647857, 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001080982 2024-03-16T11:08:58.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 16, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003802976, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000237686 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-03-16T11:08:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.854 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-03-16T11:08:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 8, 2024-03-16T11:08:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.014582633, 2024-03-16T11:08:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001822829 2024-03-16T11:08:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-03-16T11:08:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.855 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-03-16T11:08:58.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.001204939, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000602469 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003600571, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001800285 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.011634494, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.005817247 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.001020737, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001020737 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.859 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 8, 2024-03-16T11:08:58.860 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.018708536, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002338567 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 4, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.004596042, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001149010 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.861 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 3, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 2, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-03-16T11:08:58.862 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 18, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 18, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 181, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:08:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:08:58.864 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 307825, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 124, 2024-03-16T11:08:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 176, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 2, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 7624, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 160, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T11:08:58.866 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 97705, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 695, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 953, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 677201, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 30, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 953, 2024-03-16T11:08:58.867 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 953, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 104.359658829, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.109506462 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 953, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1113 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 953, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 81, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 872, 2024-03-16T11:08:58.868 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 35, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 28, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 6, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 127, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 45, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 580, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-03-16T11:08:58.869 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 75, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 56, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-03-16T11:08:58.870 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-03-16T11:08:58.871 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 90, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 3, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 12, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 12, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 149, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 0, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 12 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 1, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-03-16T11:08:58.872 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 67, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 4, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 68, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 5, 2024-03-16T11:08:58.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 2070, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1198046, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 2070, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2070, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1198046, 2024-03-16T11:08:58.874 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:58.875 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 953, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 677706, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 953, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 677706, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-03-16T11:08:58.876 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 953, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 953, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 953, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 953, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 5, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 505, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 5, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 505, 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:08:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55fa941d2500": { 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 155, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 609188, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 155, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:08:58.879 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 126, 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 609188, 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:08:58.880 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T11:08:59.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:08:59 smithi098 ceph-mon[35848]: pgmap v438: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:08:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:08:59 smithi153 ceph-mon[35521]: pgmap v438: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:08:59.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:08:59 smithi078 ceph-mon[32235]: pgmap v438: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:08:59.680 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-16T11:08:59.680 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-16T11:08:59.681 DEBUG:tasks.check_counter:Getting stats from e 2024-03-16T11:08:59.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-16T11:08:59.936 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:01 smithi098 ceph-mon[35848]: pgmap v439: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:01 smithi153 ceph-mon[35521]: pgmap v439: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:01.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:01 smithi078 ceph-mon[32235]: pgmap v439: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:01.592 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T11:09:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:09:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 106, 2024-03-16T11:09:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 99, 2024-03-16T11:09:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 739042, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 56260, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.015758534, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.003769789, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.014373704, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.001556011, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 98, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003728558, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000038046 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 738208, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 55936 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 212, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 280, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 52088, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 263151, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-16T11:09:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.018290026, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.006055088, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.004459111, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005847375, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 278, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.008529979, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000030683 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 196048 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 608, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 608, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 185408, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 192578, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.039443518, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.012871704, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.010839783, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.012740390, 2024-03-16T11:09:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 607, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.015217712, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000025070 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-03-16T11:09:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1400, 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.030095525, 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000021496 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-16T11:09:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-03-16T11:09:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 1, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-03-16T11:09:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T11:09:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-03-16T11:09:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-03-16T11:09:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T11:09:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 7, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-03-16T11:09:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 2, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 4196449, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 4196308, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 7 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 3, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 3, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 0, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 1, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 1, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 0, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 0, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 0, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 0, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-03-16T11:09:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 51912, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215420 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-03-16T11:09:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-03-16T11:09:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-03-16T11:09:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-03-16T11:09:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 3, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 2, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 2, 2024-03-16T11:09:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 2450, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:09:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:09:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 189239, 2024-03-16T11:09:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 48, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 7624, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 160, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T11:09:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 8464, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 13, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 816, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 816, 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-03-16T11:09:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 816, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.362995323, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000444847 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 816, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 817 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 816, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 816, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 507, 2024-03-16T11:09:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 307, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-03-16T11:09:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-03-16T11:09:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 90, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 2, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 5, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 5, 2024-03-16T11:09:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-03-16T11:09:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 922, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 885208, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 922, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 922, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 885208, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 816, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 2150, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 816, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2150, 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 816, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 816, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 816, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 816, 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:09:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55b7ae1266e0": { 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:09:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:01.640 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T11:09:02.389 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-16T11:09:02.389 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-16T11:09:02.389 DEBUG:tasks.check_counter:Getting stats from h 2024-03-16T11:09:02.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-16T11:09:02.636 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:09:03.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:03 smithi098 ceph-mon[35848]: pgmap v440: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:03 smithi153 ceph-mon[35521]: pgmap v440: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:03.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:03 smithi078 ceph-mon[32235]: pgmap v440: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:04.261 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T11:09:04.261 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:09:04.261 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 288, 2024-03-16T11:09:04.261 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 335, 2024-03-16T11:09:04.261 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 787059, 2024-03-16T11:09:04.261 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 164111, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.031596184, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.010580742, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.021224434, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.003109653, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 338, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.018670149, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000055237 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 322, 2024-03-16T11:09:04.262 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000104068, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000323 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 745328, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 60096 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 33, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 98, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 9312, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 326925, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 17, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 8, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.009917009, 2024-03-16T11:09:04.263 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.002696533, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.006289212, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000280459, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 97, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.020171425, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000207952 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 17, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000001818, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000106 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 4080, 2024-03-16T11:09:04.264 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 198672 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 28, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 34, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 6903, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 55100, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 12, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 5, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.004307723, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000469449, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.003324407, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000321044, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 28, 2024-03-16T11:09:04.265 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.008281989, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000295785 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 25, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000002631, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000105 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1936, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 1584 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T11:09:04.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 57, 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.001230965, 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000021595 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-16T11:09:04.267 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2, 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000005176, 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000002588 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 3, 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 1, 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1, 2024-03-16T11:09:04.268 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000157242, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000157242 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 2, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 11, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-03-16T11:09:04.269 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 11, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 0, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 11, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 10, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 2, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 5, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 2, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-03-16T11:09:04.270 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 1, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 8, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 195297298, 2024-03-16T11:09:04.271 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T11:09:04.272 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 2 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 2, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-03-16T11:09:04.273 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-03-16T11:09:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-03-16T11:09:04.275 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-03-16T11:09:04.275 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-03-16T11:09:04.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.275 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-03-16T11:09:04.275 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 7, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 7, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 2, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-03-16T11:09:04.276 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 2, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 4196449, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 0, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.009273706, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001545617 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 0 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 14, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 14, 2024-03-16T11:09:04.277 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 0, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 12, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 13, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 1, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 11, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 11, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 0, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 1, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 1, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 52484, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 223612 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-03-16T11:09:04.278 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 4, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 3, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 12, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000157242, 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000157242 2024-03-16T11:09:04.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.282 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.283 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-03-16T11:09:04.283 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.283 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.284 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.285 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-03-16T11:09:04.286 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-03-16T11:09:04.287 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-03-16T11:09:04.288 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-03-16T11:09:04.289 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-03-16T11:09:04.290 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 3, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 2, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 1, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 1, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 190, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:09:04.291 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:09:04.292 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:09:04.293 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 231047, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 79, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 7624, 2024-03-16T11:09:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 160, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 54856, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 213, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-03-16T11:09:04.295 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 43, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 12537, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 33, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 43, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 43, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.222997134, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.005185979 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-03-16T11:09:04.296 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 43, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 65 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 43, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 0, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 43, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 0, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 10, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 0, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 6, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 8, 2024-03-16T11:09:04.297 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 6, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 20, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-03-16T11:09:04.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-03-16T11:09:04.299 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 90, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 2, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 8, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 8, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 14, 2024-03-16T11:09:04.300 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 0, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 1 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 1, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-03-16T11:09:04.301 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 345, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 762026, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:04.302 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 345, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 345, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 762026, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:04.303 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 43, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 12537, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 43, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 12537, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-03-16T11:09:04.304 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 43, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 43, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 43, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 43, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-03-16T11:09:04.306 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55b401a90500": { 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 7, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 2145, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 7, 2024-03-16T11:09:04.307 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 6, 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2145, 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:04.308 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T11:09:04.980 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-16T11:09:04.980 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-16T11:09:04.980 DEBUG:tasks.check_counter:Getting stats from k 2024-03-16T11:09:04.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-16T11:09:05.258 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:09:05.422 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:05 smithi098 ceph-mon[35848]: pgmap v441: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:05.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:05 smithi153 ceph-mon[35521]: pgmap v441: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:05.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:05 smithi078 ceph-mon[32235]: pgmap v441: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-16T11:09:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.000943232, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000187956, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 5, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 73, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3152, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 90608, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.003642215, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.002071942, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001027099, 2024-03-16T11:09:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 72, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003073508, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000042687 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 90608 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:09:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 99, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 94, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 716224, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 56672, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.015969739, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.003880148, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.015257269, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.001620642, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 93, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003118529, 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000033532 2024-03-16T11:09:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 716224, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 56672 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T11:09:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:09:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:09:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148903, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T11:09:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T11:09:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 100, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 702297, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 100, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 100, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 702297, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T11:09:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T11:09:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T11:09:06.839 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T11:09:07.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:07 smithi098 ceph-mon[35848]: pgmap v442: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:09:07.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:07 smithi153 ceph-mon[35521]: pgmap v442: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:09:07.552 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-16T11:09:07.553 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-16T11:09:07.553 DEBUG:tasks.check_counter:Getting stats from c 2024-03-16T11:09:07.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-16T11:09:07.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:07 smithi078 ceph-mon[32235]: pgmap v442: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:09:07.803 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:09:09.273 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:09 smithi153 ceph-mon[35521]: pgmap v443: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:09:09.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:09 smithi098 ceph-mon[35848]: pgmap v443: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:09:09.548 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T11:09:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:09:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 701, 2024-03-16T11:09:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 703, 2024-03-16T11:09:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 814691, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 222928, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.043586065, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.012244284, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.012330123, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.014665360, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 701, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.016284322, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000023230 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 5, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 73, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1816, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 199002, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 5, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-03-16T11:09:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.004184535, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.002155424, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.001389048, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000106277, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 71, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.005289623, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000074501 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 197664 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 396, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 387, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 811813, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 147815, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.036665889, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.010690659, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.021234234, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.008805224, 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 387, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.011080092, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000028630 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 741632, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 57184 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T11:09:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1661, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.032084518, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000019316 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-16T11:09:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 21, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 4, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 17, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 14, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 2, 2024-03-16T11:09:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-16T11:09:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-16T11:09:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-16T11:09:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-16T11:09:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 155, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-16T11:09:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 13, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4729057, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4803492, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4799734, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 155 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 15, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 20, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 5, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 10, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 11, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 1, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 21, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 25, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 4, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 50944, 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-03-16T11:09:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.568 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-16T11:09:09.568 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.569 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:09 smithi078 ceph-mon[32235]: pgmap v443: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-16T11:09:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-16T11:09:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-16T11:09:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-16T11:09:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-03-16T11:09:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 3, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 2, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 2355, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:09:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:09:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:09:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 189239, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 48, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 7624, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 160, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T11:09:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 60961, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 347, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 990, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 26, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 4, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 990, 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-16T11:09:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 990, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.430992284, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000435345 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 990, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 991 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 990, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 990, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 590, 2024-03-16T11:09:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 397, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 2, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-16T11:09:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-16T11:09:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 90, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 2, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 5, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 5, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-16T11:09:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-16T11:09:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 1098, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1521052, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 1098, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1098, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1521052, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 990, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 609198, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 990, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 609198, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-16T11:09:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 990, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 990, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 990, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 990, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T11:09:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x5566208926e0": { 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:09.592 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T11:09:10.313 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-16T11:09:10.313 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-16T11:09:10.313 DEBUG:tasks.check_counter:Getting stats from f 2024-03-16T11:09:10.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-16T11:09:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:09:11.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:11 smithi153 ceph-mon[35521]: pgmap v444: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:11.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:11 smithi098 ceph-mon[35848]: pgmap v444: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:11.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:11 smithi078 ceph-mon[32235]: pgmap v444: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:12.175 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T11:09:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:09:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 823, 2024-03-16T11:09:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 825, 2024-03-16T11:09:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 237448, 2024-03-16T11:09:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 261721, 2024-03-16T11:09:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-03-16T11:09:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-16T11:09:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.050403115, 2024-03-16T11:09:12.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.015135593, 2024-03-16T11:09:12.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.013315257, 2024-03-16T11:09:12.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.017753354, 2024-03-16T11:09:12.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:12.177 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 823, 2024-03-16T11:09:12.177 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.019333080, 2024-03-16T11:09:12.178 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000023490 2024-03-16T11:09:12.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:12.178 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.178 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.178 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.179 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:09:12.179 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:09:12.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.179 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:09:12.179 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 7, 2024-03-16T11:09:12.180 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 74, 2024-03-16T11:09:12.180 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 3708, 2024-03-16T11:09:12.180 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 200602, 2024-03-16T11:09:12.180 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-03-16T11:09:12.180 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-03-16T11:09:12.180 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.004345911, 2024-03-16T11:09:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.002354645, 2024-03-16T11:09:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.001412754, 2024-03-16T11:09:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000057852, 2024-03-16T11:09:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 73, 2024-03-16T11:09:12.181 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004144488, 2024-03-16T11:09:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000056773 2024-03-16T11:09:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.182 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:12.183 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.183 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.183 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.183 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T11:09:12.183 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 199904 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 108, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 101, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 739020, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 57534, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.015726929, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.003896724, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.014723562, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.001681993, 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:12.184 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 100, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004035684, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000040356 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 738752, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 57216 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T11:09:12.185 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1442, 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.027813159, 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000019287 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.187 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.188 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 0, 2024-03-16T11:09:12.189 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-03-16T11:09:12.190 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 0, 2024-03-16T11:09:12.190 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-16T11:09:12.190 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 0, 2024-03-16T11:09:12.190 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-16T11:09:12.190 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-03-16T11:09:12.190 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-03-16T11:09:12.191 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 1, 2024-03-16T11:09:12.191 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-03-16T11:09:12.191 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-03-16T11:09:12.191 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-16T11:09:12.191 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-03-16T11:09:12.191 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-16T11:09:12.191 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-16T11:09:12.192 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-16T11:09:12.192 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-16T11:09:12.192 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-16T11:09:12.192 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-16T11:09:12.192 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-16T11:09:12.192 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-16T11:09:12.192 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-16T11:09:12.193 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-16T11:09:12.193 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-16T11:09:12.193 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-16T11:09:12.193 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-03-16T11:09:12.193 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-03-16T11:09:12.193 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-16T11:09:12.193 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-16T11:09:12.194 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-16T11:09:12.195 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T11:09:12.195 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-16T11:09:12.195 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T11:09:12.195 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T11:09:12.195 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T11:09:12.195 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-03-16T11:09:12.196 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-16T11:09:12.196 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-16T11:09:12.196 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-03-16T11:09:12.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.196 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-16T11:09:12.196 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-16T11:09:12.197 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-16T11:09:12.197 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-16T11:09:12.197 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-03-16T11:09:12.197 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-16T11:09:12.197 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-16T11:09:12.197 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-03-16T11:09:12.198 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 5, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-16T11:09:12.199 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4195917, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4195776, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 5 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 3, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 3, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 1, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 1, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 0, 2024-03-16T11:09:12.201 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 52792, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-16T11:09:12.202 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.203 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.207 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.207 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-16T11:09:12.207 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.207 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.208 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.209 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.210 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.214 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.216 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-16T11:09:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-16T11:09:12.218 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 2, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 1, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 2357, 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-16T11:09:12.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:09:12.220 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:09:12.221 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 181159, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 44, 2024-03-16T11:09:12.222 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 7624, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 160, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 8464, 2024-03-16T11:09:12.223 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 13, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 826, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 826, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 826, 2024-03-16T11:09:12.224 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.399992839, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000484252 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 826, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 827 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 826, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 826, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 508, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-03-16T11:09:12.225 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 316, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-16T11:09:12.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-16T11:09:12.227 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 90, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 2, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 4, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 4, 2024-03-16T11:09:12.228 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.229 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 934, 2024-03-16T11:09:12.230 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 887662, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 934, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 934, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 887662, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:12.231 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 826, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 1618, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 826, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1618, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.232 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 826, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 826, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 826, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 826, 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:12.233 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:12.234 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x565209d686e0": { 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:12.235 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:12.236 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T11:09:12.985 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-16T11:09:12.986 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-16T11:09:12.986 DEBUG:tasks.check_counter:Getting stats from i 2024-03-16T11:09:12.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-16T11:09:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:09:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:13 smithi153 ceph-mon[35521]: pgmap v445: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:13.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:13 smithi098 ceph-mon[35848]: pgmap v445: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:13.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:13 smithi078 ceph-mon[32235]: pgmap v445: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:14.948 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T11:09:14.949 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:09:14.949 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 187, 2024-03-16T11:09:14.949 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 244, 2024-03-16T11:09:14.949 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 52176, 2024-03-16T11:09:14.949 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 118070, 2024-03-16T11:09:14.950 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 10, 2024-03-16T11:09:14.950 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-03-16T11:09:14.950 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.018563028, 2024-03-16T11:09:14.950 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.005598083, 2024-03-16T11:09:14.950 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.009468998, 2024-03-16T11:09:14.950 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.001650561, 2024-03-16T11:09:14.951 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:14.951 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 243, 2024-03-16T11:09:14.951 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.020532094, 2024-03-16T11:09:14.951 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000084494 2024-03-16T11:09:14.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.951 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:14.952 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 367, 2024-03-16T11:09:14.952 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000070999, 2024-03-16T11:09:14.952 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000193 2024-03-16T11:09:14.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.952 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-16T11:09:14.952 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T11:09:14.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.953 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:09:14.953 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 160, 2024-03-16T11:09:14.953 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 150, 2024-03-16T11:09:14.953 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 755320, 2024-03-16T11:09:14.953 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 116080, 2024-03-16T11:09:14.954 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 10, 2024-03-16T11:09:14.954 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-03-16T11:09:14.954 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.019498762, 2024-03-16T11:09:14.954 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.004445547, 2024-03-16T11:09:14.954 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.017999276, 2024-03-16T11:09:14.954 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.001834442, 2024-03-16T11:09:14.955 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:14.955 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 149, 2024-03-16T11:09:14.955 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.016807141, 2024-03-16T11:09:14.955 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000112799 2024-03-16T11:09:14.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.955 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:14.956 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 26, 2024-03-16T11:09:14.956 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000003370, 2024-03-16T11:09:14.956 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000129 2024-03-16T11:09:14.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.956 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 749200, 2024-03-16T11:09:14.956 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 60736 2024-03-16T11:09:14.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.957 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:09:14.957 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 27, 2024-03-16T11:09:14.957 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 95, 2024-03-16T11:09:14.957 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 6347, 2024-03-16T11:09:14.957 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 305112, 2024-03-16T11:09:14.958 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 15, 2024-03-16T11:09:14.958 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 6, 2024-03-16T11:09:14.958 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.008250022, 2024-03-16T11:09:14.958 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.002641740, 2024-03-16T11:09:14.958 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.004766424, 2024-03-16T11:09:14.958 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000268085, 2024-03-16T11:09:14.959 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:14.959 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 93, 2024-03-16T11:09:14.959 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.009682519, 2024-03-16T11:09:14.959 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000104113 2024-03-16T11:09:14.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.959 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:14.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 25, 2024-03-16T11:09:14.960 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000002359, 2024-03-16T11:09:14.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000094 2024-03-16T11:09:14.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.960 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 2672, 2024-03-16T11:09:14.961 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 202752 2024-03-16T11:09:14.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.961 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T11:09:14.961 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-16T11:09:14.961 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T11:09:14.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.962 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-16T11:09:14.962 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T11:09:14.962 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T11:09:14.962 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 45, 2024-03-16T11:09:14.962 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.002519248, 2024-03-16T11:09:14.962 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000055983 2024-03-16T11:09:14.963 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:14.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.963 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-16T11:09:14.963 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T11:09:14.963 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T11:09:14.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T11:09:14.964 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000015078, 2024-03-16T11:09:14.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000007539 2024-03-16T11:09:14.964 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:14.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.964 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-16T11:09:14.965 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 10, 2024-03-16T11:09:14.965 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 7, 2024-03-16T11:09:14.965 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-16T11:09:14.965 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 7, 2024-03-16T11:09:14.965 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.002149577, 2024-03-16T11:09:14.965 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000307082 2024-03-16T11:09:14.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.966 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-16T11:09:14.966 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 3, 2024-03-16T11:09:14.966 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-16T11:09:14.966 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-16T11:09:14.966 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-03-16T11:09:14.967 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-16T11:09:14.967 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-16T11:09:14.967 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 14, 2024-03-16T11:09:14.967 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-03-16T11:09:14.967 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 14, 2024-03-16T11:09:14.967 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-16T11:09:14.967 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 14, 2024-03-16T11:09:14.968 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-16T11:09:14.968 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 4, 2024-03-16T11:09:14.968 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 4, 2024-03-16T11:09:14.968 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 2, 2024-03-16T11:09:14.968 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 12, 2024-03-16T11:09:14.968 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 10, 2024-03-16T11:09:14.968 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 1, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-16T11:09:14.969 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 8, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 195297298, 2024-03-16T11:09:14.970 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-16T11:09:14.971 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T11:09:14.971 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-16T11:09:14.971 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T11:09:14.971 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-16T11:09:14.971 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-16T11:09:14.971 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T11:09:14.971 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 1, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 1, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 1 2024-03-16T11:09:14.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 2, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-16T11:09:14.973 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T11:09:14.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 5, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 5, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-16T11:09:14.975 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4195917, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.005697857, 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001424464 2024-03-16T11:09:14.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 17, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 17, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 15, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 16, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 1, 2024-03-16T11:09:14.977 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 14, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 14, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 4, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 7, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 3, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 54844, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.978 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 11, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 10, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 9, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 5, 2024-03-16T11:09:14.979 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001837751, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000367550 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-16T11:09:14.980 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000311826, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000155913 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.981 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-16T11:09:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-16T11:09:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.986 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-16T11:09:14.986 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.986 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.986 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.986 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-16T11:09:14.987 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.987 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.987 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.987 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-16T11:09:14.987 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.987 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.988 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.989 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.990 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.991 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:14.992 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 2, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 1, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 6, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 3, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 198, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T11:09:14.993 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:09:14.994 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:09:14.995 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:09:14.996 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 248279, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 92, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 176, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 2, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T11:09:14.997 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 7624, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 160, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:09:14.998 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 68925, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 301, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:14.999 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 41, 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 12196, 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 29, 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 41, 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 41, 2024-03-16T11:09:15.000 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.214996131, 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.005243808 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 41, 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 64 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 41, 2024-03-16T11:09:15.001 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 0, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 41, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 0, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 4, 2024-03-16T11:09:15.002 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 7, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 6, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-16T11:09:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T11:09:15.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-16T11:09:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-16T11:09:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 90, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 2, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 9, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 9, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-16T11:09:15.007 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 16, 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 1 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-16T11:09:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-16T11:09:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 370, 2024-03-16T11:09:15.010 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 770398, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 370, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 370, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 770398, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:15.011 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 41, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 12196, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 41, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 12196, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:15.012 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 41, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 41, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 41, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 41, 2024-03-16T11:09:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T11:09:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x556c95730500": { 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 5, 2024-03-16T11:09:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1613, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 5, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 4, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1613, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:15.016 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T11:09:15.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:15 smithi153 ceph-mon[35521]: pgmap v446: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:15.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:15 smithi098 ceph-mon[35848]: pgmap v446: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:15.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:15 smithi078 ceph-mon[32235]: pgmap v446: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:15.627 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-16T11:09:15.627 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-16T11:09:15.627 DEBUG:tasks.check_counter:Getting stats from l 2024-03-16T11:09:15.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid a1941e5e-e383-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-16T11:09:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:09:17.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:17 smithi153 ceph-mon[35521]: pgmap v447: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:09:17.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:17 smithi098 ceph-mon[35848]: pgmap v447: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:09:17.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:17 smithi078 ceph-mon[32235]: pgmap v447: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-16T11:09:17.593 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T11:09:17.593 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 0, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 0, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 0, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 0, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.001110322, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.000274088, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:17.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 107, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 100, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 720304, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 59680, 2024-03-16T11:09:17.595 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 2, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.016299544, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.003935135, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.015436976, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.001772882, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 100, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004365660, 2024-03-16T11:09:17.596 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000043656 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 720304, 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 59680 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T11:09:17.597 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 74, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 160, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 90688, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.003299795, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.002137915, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.000550269, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 72, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.003510972, 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000048763 2024-03-16T11:09:17.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 90688 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 0, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T11:09:17.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T11:09:17.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T11:09:17.601 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 148863, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 28, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T11:09:17.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 928, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 2, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T11:09:17.603 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 104, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 702817, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 104, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 104, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 702817, 2024-03-16T11:09:17.604 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T11:09:17.605 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T11:09:17.605 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T11:09:17.605 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T11:09:17.605 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:17.605 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T11:09:17.605 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T11:09:18.334 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-16T11:09:18.334 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-16T11:09:18.334 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-16T11:09:18.345 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-16T11:09:18.345 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T11:09:18.346 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T11:09:18.368 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T11:09:18.368 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T11:09:18.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:09:18.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 -- 172.21.15.78:0/2037681998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72941018f0 msgr2=0x7f7294101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2037681998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72941018f0 0x7f7294101d70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7288009900 tx=0x7f728802bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:18.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 -- 172.21.15.78:0/2037681998 shutdown_connections 2024-03-16T11:09:18.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2037681998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72941022b0 0x7f7294108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2037681998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72941018f0 0x7f7294101d70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2037681998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7294100700 0x7f7294100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 -- 172.21.15.78:0/2037681998 >> 172.21.15.78:0/2037681998 conn(0x7f72940fbc90 msgr2=0x7f72940fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.586+0000 7f729a5cd640 1 -- 172.21.15.78:0/2037681998 shutdown_connections 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 -- 172.21.15.78:0/2037681998 wait complete. 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 Processor -- start 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 -- start start 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7294100700 0x7f7294196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72941018f0 0x7f7294196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72941022b0 0x7f729419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f729410c230 con 0x7f7294100700 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f729410c0b0 con 0x7f72941022b0 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.587+0000 7f729a5cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f729410c3b0 con 0x7f72941018f0 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7294100700 0x7f7294196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7294100700 0x7f7294196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52316/0 (socket says 172.21.15.78:52316) 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f72937fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72941018f0 0x7f7294196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 -- 172.21.15.78:0/2830706671 learned_addr learned my addr 172.21.15.78:0/2830706671 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7298b6a640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72941022b0 0x7f729419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 -- 172.21.15.78:0/2830706671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72941018f0 msgr2=0x7f7294196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72941018f0 0x7f7294196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 -- 172.21.15.78:0/2830706671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72941022b0 msgr2=0x7f729419df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72941022b0 0x7f729419df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 -- 172.21.15.78:0/2830706671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729419e570 con 0x7f7294100700 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f72937fe640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72941018f0 0x7f7294196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f7293fff640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7294100700 0x7f7294196950 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f728400e9c0 tx=0x7f728400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f72917fa640 1 -- 172.21.15.78:0/2830706671 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728401b070 con 0x7f7294100700 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f72917fa640 1 -- 172.21.15.78:0/2830706671 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7284004030 con 0x7f7294100700 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7294069e10 con 0x7f7294100700 2024-03-16T11:09:18.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f72917fa640 1 -- 172.21.15.78:0/2830706671 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284003c30 con 0x7f7294100700 2024-03-16T11:09:18.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.588+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729406a2e0 con 0x7f7294100700 2024-03-16T11:09:18.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.589+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7294068a20 con 0x7f7294100700 2024-03-16T11:09:18.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.591+0000 7f72917fa640 1 -- 172.21.15.78:0/2830706671 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f728401e070 con 0x7f7294100700 2024-03-16T11:09:18.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.592+0000 7f72917fa640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f727c0a2800 0x7f727c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.592+0000 7f72917fa640 1 -- 172.21.15.78:0/2830706671 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f72840ce6d0 con 0x7f7294100700 2024-03-16T11:09:18.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.592+0000 7f72937fe640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f727c0a2800 0x7f727c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.592+0000 7f72937fe640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f727c0a2800 0x7f727c0a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7288009230 tx=0x7f7288005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:18.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.596+0000 7f72917fa640 1 -- 172.21.15.78:0/2830706671 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7284025020 con 0x7f7294100700 2024-03-16T11:09:18.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.776+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7294107560 con 0x7f7294100700 2024-03-16T11:09:18.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.777+0000 7f72917fa640 1 -- 172.21.15.78:0/2830706671 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f728409b120 con 0x7f7294100700 2024-03-16T11:09:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:09:18.778 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:09:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.779+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f727c0a2800 msgr2=0x7f727c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.779+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f727c0a2800 0x7f727c0a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7288009230 tx=0x7f7288005fd0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.779+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7294100700 msgr2=0x7f7294196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.779+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7294100700 0x7f7294196950 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f728400e9c0 tx=0x7f728400ee80 comp rx=0 tx=0).stop 2024-03-16T11:09:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 shutdown_connections 2024-03-16T11:09:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72941022b0 0x7f729419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f727c0a2800 0x7f727c0a4cc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72941018f0 0x7f7294196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 --2- 172.21.15.78:0/2830706671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7294100700 0x7f7294196950 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 >> 172.21.15.78:0/2830706671 conn(0x7f72940fbc90 msgr2=0x7f7294104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:18.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 shutdown_connections 2024-03-16T11:09:18.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.780+0000 7f729a5cd640 1 -- 172.21.15.78:0/2830706671 wait complete. 2024-03-16T11:09:18.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:09:18.792 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 -- 172.21.15.78:0/2404619170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 msgr2=0x7f1be8101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 --2- 172.21.15.78:0/2404619170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8101c10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009900 tx=0x7f1bd802bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 -- 172.21.15.78:0/2404619170 shutdown_connections 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 --2- 172.21.15.78:0/2404619170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1be81033a0 0x7f1be8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 --2- 172.21.15.78:0/2404619170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 0x7f1be8102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 --2- 172.21.15.78:0/2404619170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8101c10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 -- 172.21.15.78:0/2404619170 >> 172.21.15.78:0/2404619170 conn(0x7f1be80fcd40 msgr2=0x7f1be80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 -- 172.21.15.78:0/2404619170 shutdown_connections 2024-03-16T11:09:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.899+0000 7f1bee64b640 1 -- 172.21.15.78:0/2404619170 wait complete. 2024-03-16T11:09:18.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 Processor -- start 2024-03-16T11:09:18.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 -- start start 2024-03-16T11:09:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 0x7f1be8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1be81033a0 0x7f1be819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1be810d320 con 0x7f1be81029e0 2024-03-16T11:09:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1be810d1a0 con 0x7f1be81017f0 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1bee64b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1be810d4a0 con 0x7f1be81033a0 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 0x7f1be8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.78:36352/0 (socket says 172.21.15.78:36352) 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 0x7f1be8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52340/0 (socket says 172.21.15.78:52340) 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 -- 172.21.15.78:0/3980387408 learned_addr learned my addr 172.21.15.78:0/3980387408 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1becbe8640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1be81033a0 0x7f1be819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 -- 172.21.15.78:0/3980387408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1be81033a0 msgr2=0x7f1be819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1be81033a0 0x7f1be819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 -- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 msgr2=0x7f1be8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 0x7f1be8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be7fff640 1 -- 172.21.15.78:0/3980387408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be819f690 con 0x7f1be81017f0 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.900+0000 7f1be77fe640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 0x7f1be8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.901+0000 7f1be7fff640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8197ac0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8000c00 tx=0x7f1bd8005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.901+0000 7f1be57fa640 1 -- 172.21.15.78:0/3980387408 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8043020 con 0x7f1be81017f0 2024-03-16T11:09:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.901+0000 7f1be57fa640 1 -- 172.21.15.78:0/3980387408 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bd8004350 con 0x7f1be81017f0 2024-03-16T11:09:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.901+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be819f920 con 0x7f1be81017f0 2024-03-16T11:09:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.901+0000 7f1be57fa640 1 -- 172.21.15.78:0/3980387408 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8003b80 con 0x7f1be81017f0 2024-03-16T11:09:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.901+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be819fe70 con 0x7f1be81017f0 2024-03-16T11:09:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.902+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be8108c80 con 0x7f1be81017f0 2024-03-16T11:09:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.903+0000 7f1be57fa640 1 -- 172.21.15.78:0/3980387408 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1bd8046020 con 0x7f1be81017f0 2024-03-16T11:09:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.903+0000 7f1be57fa640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1bd00a2800 0x7f1bd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:18.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.904+0000 7f1be77fe640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1bd00a2800 0x7f1bd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:18.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.904+0000 7f1be57fa640 1 -- 172.21.15.78:0/3980387408 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1bd80f0f40 con 0x7f1be81017f0 2024-03-16T11:09:18.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.904+0000 7f1be77fe640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1bd00a2800 0x7f1bd00a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1be8198f80 tx=0x7f1bdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:18.907+0000 7f1be57fa640 1 -- 172.21.15.78:0/3980387408 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1bd804d020 con 0x7f1be81017f0 2024-03-16T11:09:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.091+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1be802d1b0 con 0x7f1be81017f0 2024-03-16T11:09:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.092+0000 7f1be57fa640 1 -- 172.21.15.78:0/3980387408 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f1bd80bdfe0 con 0x7f1be81017f0 2024-03-16T11:09:19.092 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:09:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1bd00a2800 msgr2=0x7f1bd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1bd00a2800 0x7f1bd00a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1be8198f80 tx=0x7f1bdc005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 msgr2=0x7f1be8197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8197ac0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8000c00 tx=0x7f1bd8005dc0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 shutdown_connections 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1bd00a2800 0x7f1bd00a4cc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1be81033a0 0x7f1be819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be81029e0 0x7f1be8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 --2- 172.21.15.78:0/3980387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1be81017f0 0x7f1be8197ac0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 >> 172.21.15.78:0/3980387408 conn(0x7f1be80fcd40 msgr2=0x7f1be8105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 shutdown_connections 2024-03-16T11:09:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.094+0000 7f1bee64b640 1 -- 172.21.15.78:0/3980387408 wait complete. 2024-03-16T11:09:19.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:09:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.216+0000 7f62ae59e640 1 -- 172.21.15.78:0/1899821251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a81018d0 msgr2=0x7f62a8101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.216+0000 7f62ae59e640 1 --2- 172.21.15.78:0/1899821251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a81018d0 0x7f62a8101d50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f629c009900 tx=0x7f629c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.216+0000 7f62ae59e640 1 -- 172.21.15.78:0/1899821251 shutdown_connections 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.216+0000 7f62ae59e640 1 --2- 172.21.15.78:0/1899821251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a8102290 0x7f62a8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.216+0000 7f62ae59e640 1 --2- 172.21.15.78:0/1899821251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a81018d0 0x7f62a8101d50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.216+0000 7f62ae59e640 1 --2- 172.21.15.78:0/1899821251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 0x7f62a8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.216+0000 7f62ae59e640 1 -- 172.21.15.78:0/1899821251 >> 172.21.15.78:0/1899821251 conn(0x7f62a80fbc70 msgr2=0x7f62a80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 -- 172.21.15.78:0/1899821251 shutdown_connections 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 -- 172.21.15.78:0/1899821251 wait complete. 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 Processor -- start 2024-03-16T11:09:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 -- start start 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 0x7f62a80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81018d0 0x7f62a819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8102290 0x7f62a819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f62a810c1f0 con 0x7f62a8102290 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62a810c070 con 0x7f62a81018d0 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.217+0000 7f62ae59e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62a810c370 con 0x7f62a81006e0 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 0x7f62a80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62acb3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8102290 0x7f62a819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 0x7f62a80694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.78:38306/0 (socket says 172.21.15.78:38306) 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a7fff640 1 -- 172.21.15.78:0/302250303 learned_addr learned my addr 172.21.15.78:0/302250303 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a77fe640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81018d0 0x7f62a819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62acb3b640 1 -- 172.21.15.78:0/302250303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 msgr2=0x7f62a80694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62acb3b640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 0x7f62a80694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62acb3b640 1 -- 172.21.15.78:0/302250303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81018d0 msgr2=0x7f62a819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62acb3b640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81018d0 0x7f62a819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62acb3b640 1 -- 172.21.15.78:0/302250303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a819f6b0 con 0x7f62a8102290 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a7fff640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 0x7f62a80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62acb3b640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8102290 0x7f62a819eda0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009fd0 tx=0x7f62a000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a57fa640 1 -- 172.21.15.78:0/302250303 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a000ce20 con 0x7f62a8102290 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a57fa640 1 -- 172.21.15.78:0/302250303 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a0005020 con 0x7f62a8102290 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a819f940 con 0x7f62a8102290 2024-03-16T11:09:19.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62a57fa640 1 -- 172.21.15.78:0/302250303 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0010690 con 0x7f62a8102290 2024-03-16T11:09:19.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.218+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a819fe60 con 0x7f62a8102290 2024-03-16T11:09:19.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.219+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a8107b20 con 0x7f62a8102290 2024-03-16T11:09:19.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.220+0000 7f62a57fa640 1 -- 172.21.15.78:0/302250303 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f62a00051d0 con 0x7f62a8102290 2024-03-16T11:09:19.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.221+0000 7f62a57fa640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62900a2880 0x7f62900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.221+0000 7f62a7fff640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62900a2880 0x7f62900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.221+0000 7f62a57fa640 1 -- 172.21.15.78:0/302250303 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f62a00cefe0 con 0x7f62a8102290 2024-03-16T11:09:19.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.221+0000 7f62a7fff640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62900a2880 0x7f62900a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f62980058c0 tx=0x7f6298005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.225+0000 7f62a57fa640 1 -- 172.21.15.78:0/302250303 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62a009c120 con 0x7f62a8102290 2024-03-16T11:09:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.402+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f62a80623e0 con 0x7f62a8102290 2024-03-16T11:09:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.403+0000 7f62a57fa640 1 -- 172.21.15.78:0/302250303 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f62a009ba20 con 0x7f62a8102290 2024-03-16T11:09:19.404 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62900a2880 msgr2=0x7f62900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62900a2880 0x7f62900a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f62980058c0 tx=0x7f6298005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8102290 msgr2=0x7f62a819eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8102290 0x7f62a819eda0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009fd0 tx=0x7f62a000ef10 comp rx=0 tx=0).stop 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 shutdown_connections 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8102290 0x7f62a819eda0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62a81018d0 0x7f62a819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f62900a2880 0x7f62900a4d40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 --2- 172.21.15.78:0/302250303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a81006e0 0x7f62a80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 >> 172.21.15.78:0/302250303 conn(0x7f62a80fbc70 msgr2=0x7f62a8104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 shutdown_connections 2024-03-16T11:09:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.405+0000 7f62ae59e640 1 -- 172.21.15.78:0/302250303 wait complete. 2024-03-16T11:09:19.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:19 smithi078 ceph-mon[32235]: pgmap v448: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:09:19.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:19 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:09:19.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:19 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2830706671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:19.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:19 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3980387408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:09:19.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:09:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:19 smithi098 ceph-mon[35848]: pgmap v448: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:09:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:19 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:09:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:19 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2830706671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:19.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:19 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3980387408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:09:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 -- 172.21.15.78:0/3482986722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441033a0 msgr2=0x7f0a44109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 --2- 172.21.15.78:0/3482986722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441033a0 0x7f0a44109c30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009ca0 tx=0x7f0a3c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 -- 172.21.15.78:0/3482986722 shutdown_connections 2024-03-16T11:09:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 --2- 172.21.15.78:0/3482986722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441033a0 0x7f0a44109c30 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 --2- 172.21.15.78:0/3482986722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a441029e0 0x7f0a44102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 --2- 172.21.15.78:0/3482986722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a441017f0 0x7f0a44101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 -- 172.21.15.78:0/3482986722 >> 172.21.15.78:0/3482986722 conn(0x7f0a440fcd80 msgr2=0x7f0a440ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 -- 172.21.15.78:0/3482986722 shutdown_connections 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.520+0000 7f0a49910640 1 -- 172.21.15.78:0/3482986722 wait complete. 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 Processor -- start 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 -- start start 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441017f0 0x7f0a441979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a441029e0 0x7f0a44197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a441033a0 0x7f0a4419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4410d320 con 0x7f0a441017f0 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4410d1a0 con 0x7f0a441029e0 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a49910640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4410d4a0 con 0x7f0a441033a0 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441017f0 0x7f0a441979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441017f0 0x7f0a441979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52388/0 (socket says 172.21.15.78:52388) 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 -- 172.21.15.78:0/2122992392 learned_addr learned my addr 172.21.15.78:0/2122992392 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a3bfff640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a441029e0 0x7f0a44197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a437fe640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a441033a0 0x7f0a4419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 -- 172.21.15.78:0/2122992392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a441033a0 msgr2=0x7f0a4419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a441033a0 0x7f0a4419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 -- 172.21.15.78:0/2122992392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a441029e0 msgr2=0x7f0a44197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a441029e0 0x7f0a44197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 -- 172.21.15.78:0/2122992392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4419f610 con 0x7f0a441017f0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.521+0000 7f0a42ffd640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441017f0 0x7f0a441979f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400e9c0 tx=0x7f0a3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.522+0000 7f0a4890e640 1 -- 172.21.15.78:0/2122992392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3400cd80 con 0x7f0a441017f0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.522+0000 7f0a4890e640 1 -- 172.21.15.78:0/2122992392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a340035f0 con 0x7f0a441017f0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.522+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4419f900 con 0x7f0a441017f0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.522+0000 7f0a4890e640 1 -- 172.21.15.78:0/2122992392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a34010690 con 0x7f0a441017f0 2024-03-16T11:09:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.522+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4419fe00 con 0x7f0a441017f0 2024-03-16T11:09:19.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.523+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a44108c30 con 0x7f0a441017f0 2024-03-16T11:09:19.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.524+0000 7f0a4890e640 1 -- 172.21.15.78:0/2122992392 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0a340037a0 con 0x7f0a441017f0 2024-03-16T11:09:19.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.525+0000 7f0a4890e640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0a300a2800 0x7f0a300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.525+0000 7f0a4890e640 1 -- 172.21.15.78:0/2122992392 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0a340cf0d0 con 0x7f0a441017f0 2024-03-16T11:09:19.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.525+0000 7f0a3bfff640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0a300a2800 0x7f0a300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.525+0000 7f0a3bfff640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0a300a2800 0x7f0a300a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0a44198f00 tx=0x7f0a2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:19.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.529+0000 7f0a4890e640 1 -- 172.21.15.78:0/2122992392 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a3409c170 con 0x7f0a441017f0 2024-03-16T11:09:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:19 smithi153 ceph-mon[35521]: pgmap v448: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:09:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:19 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:09:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:19 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2830706671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:19 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3980387408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:09:19.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.720+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a4402d360 con 0x7f0a441017f0 2024-03-16T11:09:19.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.721+0000 7f0a4890e640 1 -- 172.21.15.78:0/2122992392 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f0a3409ba70 con 0x7f0a441017f0 2024-03-16T11:09:19.721 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:09:19.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0a300a2800 msgr2=0x7f0a300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0a300a2800 0x7f0a300a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0a44198f00 tx=0x7f0a2c005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441017f0 msgr2=0x7f0a441979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441017f0 0x7f0a441979f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400e9c0 tx=0x7f0a3400ee80 comp rx=0 tx=0).stop 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 shutdown_connections 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f0a300a2800 0x7f0a300a4cc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a441033a0 0x7f0a4419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a441029e0 0x7f0a44197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 --2- 172.21.15.78:0/2122992392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a441017f0 0x7f0a441979f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 >> 172.21.15.78:0/2122992392 conn(0x7f0a440fcd80 msgr2=0x7f0a44105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 shutdown_connections 2024-03-16T11:09:19.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.723+0000 7f0a49910640 1 -- 172.21.15.78:0/2122992392 wait complete. 2024-03-16T11:09:19.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 -- 172.21.15.78:0/3691078362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c101800 msgr2=0x7f216c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 --2- 172.21.15.78:0/3691078362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c101800 0x7f216c101c20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f215c009900 tx=0x7f215c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 -- 172.21.15.78:0/3691078362 shutdown_connections 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 --2- 172.21.15.78:0/3691078362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c1033b0 0x7f216c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 --2- 172.21.15.78:0/3691078362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f216c1029f0 0x7f216c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 --2- 172.21.15.78:0/3691078362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c101800 0x7f216c101c20 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 -- 172.21.15.78:0/3691078362 >> 172.21.15.78:0/3691078362 conn(0x7f216c0fcd50 msgr2=0x7f216c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 -- 172.21.15.78:0/3691078362 shutdown_connections 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 -- 172.21.15.78:0/3691078362 wait complete. 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 Processor -- start 2024-03-16T11:09:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.836+0000 7f2172426640 1 -- start start 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f2172426640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f216c101800 0x7f216c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f2172426640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c1029f0 0x7f216c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f2172426640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c1033b0 0x7f216c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f2172426640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f216c10d320 con 0x7f216c1029f0 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f2172426640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f216c10d1a0 con 0x7f216c1033b0 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f2172426640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f216c10d4a0 con 0x7f216c101800 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c1029f0 0x7f216c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c1029f0 0x7f216c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52408/0 (socket says 172.21.15.78:52408) 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 -- 172.21.15.78:0/4029624947 learned_addr learned my addr 172.21.15.78:0/4029624947 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f21709c3640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c1033b0 0x7f216c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216bfff640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f216c101800 0x7f216c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 -- 172.21.15.78:0/4029624947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f216c101800 msgr2=0x7f216c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f216c101800 0x7f216c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 -- 172.21.15.78:0/4029624947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c1033b0 msgr2=0x7f216c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c1033b0 0x7f216c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 -- 172.21.15.78:0/4029624947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216c19f680 con 0x7f216c1029f0 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f216b7fe640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c1029f0 0x7f216c197fa0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.837+0000 7f21697fa640 1 -- 172.21.15.78:0/4029624947 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216000cd80 con 0x7f216c1029f0 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.838+0000 7f21697fa640 1 -- 172.21.15.78:0/4029624947 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21600054d0 con 0x7f216c1029f0 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.838+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216c19f970 con 0x7f216c1029f0 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.838+0000 7f21697fa640 1 -- 172.21.15.78:0/4029624947 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2160010490 con 0x7f216c1029f0 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.838+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216c19fe70 con 0x7f216c1029f0 2024-03-16T11:09:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.838+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f216c108c30 con 0x7f216c1029f0 2024-03-16T11:09:19.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.839+0000 7f21697fa640 1 -- 172.21.15.78:0/4029624947 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2160005020 con 0x7f216c1029f0 2024-03-16T11:09:19.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.840+0000 7f21697fa640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f21540a2800 0x7f21540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:19.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.840+0000 7f216bfff640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f21540a2800 0x7f21540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:19.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.840+0000 7f21697fa640 1 -- 172.21.15.78:0/4029624947 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f21600cedc0 con 0x7f216c1029f0 2024-03-16T11:09:19.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.840+0000 7f216bfff640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f21540a2800 0x7f21540a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f215c005ab0 tx=0x7f215c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:19.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:19.844+0000 7f21697fa640 1 -- 172.21.15.78:0/4029624947 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f216009be60 con 0x7f216c1029f0 2024-03-16T11:09:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.028+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f216c02d1b0 con 0x7f216c1029f0 2024-03-16T11:09:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.029+0000 7f21697fa640 1 -- 172.21.15.78:0/4029624947 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+591 (secure 0 0 0) 0x7f216009b760 con 0x7f216c1029f0 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/524003541 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2003716687 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3269664941 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2003716687 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1201370297 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2103714756 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1410130144 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3269664941 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2522710278 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:20.030 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4255056165 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-03-16T11:09:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f21540a2800 msgr2=0x7f21540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f21540a2800 0x7f21540a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f215c005ab0 tx=0x7f215c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c1029f0 msgr2=0x7f216c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c1029f0 0x7f216c197fa0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000edf0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 shutdown_connections 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f216c1033b0 0x7f216c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f216c1029f0 0x7f216c197fa0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f21540a2800 0x7f21540a4cc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 --2- 172.21.15.78:0/4029624947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f216c101800 0x7f216c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 >> 172.21.15.78:0/4029624947 conn(0x7f216c0fcd50 msgr2=0x7f216c105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 shutdown_connections 2024-03-16T11:09:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.031+0000 7f2172426640 1 -- 172.21.15.78:0/4029624947 wait complete. 2024-03-16T11:09:20.043 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-16T11:09:20.043 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T11:09:20.058 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:09:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 -- 172.21.15.78:0/296262574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c0102a20 msgr2=0x7f87c0102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 --2- 172.21.15.78:0/296262574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c0102a20 0x7f87c0102ea0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009900 tx=0x7f87b402bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 -- 172.21.15.78:0/296262574 shutdown_connections 2024-03-16T11:09:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 --2- 172.21.15.78:0/296262574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c01033e0 0x7f87c0109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 --2- 172.21.15.78:0/296262574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c0102a20 0x7f87c0102ea0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 --2- 172.21.15.78:0/296262574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c0101830 0x7f87c0101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 -- 172.21.15.78:0/296262574 >> 172.21.15.78:0/296262574 conn(0x7f87c00fcd80 msgr2=0x7f87c00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 -- 172.21.15.78:0/296262574 shutdown_connections 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.199+0000 7f87c50b2640 1 -- 172.21.15.78:0/296262574 wait complete. 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 Processor -- start 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 -- start start 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c0101830 0x7f87c0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c0102a20 0x7f87c0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c01033e0 0x7f87c019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f87c010d360 con 0x7f87c01033e0 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87c010d1e0 con 0x7f87c0101830 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.200+0000 7f87c50b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87c010d4e0 con 0x7f87c0102a20 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c01033e0 0x7f87c019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c01033e0 0x7f87c019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52432/0 (socket says 172.21.15.78:52432) 2024-03-16T11:09:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 -- 172.21.15.78:0/4087892464 learned_addr learned my addr 172.21.15.78:0/4087892464 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bed9d640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c0101830 0x7f87c0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87be59c640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c0102a20 0x7f87c0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 -- 172.21.15.78:0/4087892464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c0102a20 msgr2=0x7f87c0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c0102a20 0x7f87c0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 -- 172.21.15.78:0/4087892464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c0101830 msgr2=0x7f87c0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c0101830 0x7f87c0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 -- 172.21.15.78:0/4087892464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c019f650 con 0x7f87c01033e0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87bf59e640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c01033e0 0x7f87c019efe0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f87b000eb10 tx=0x7f87b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87a7fff640 1 -- 172.21.15.78:0/4087892464 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0004b20 con 0x7f87c01033e0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87a7fff640 1 -- 172.21.15.78:0/4087892464 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87b0004cc0 con 0x7f87c01033e0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c019f940 con 0x7f87c01033e0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87a7fff640 1 -- 172.21.15.78:0/4087892464 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0010690 con 0x7f87c01033e0 2024-03-16T11:09:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.201+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c019fe60 con 0x7f87c01033e0 2024-03-16T11:09:20.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.202+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c0108bd0 con 0x7f87c01033e0 2024-03-16T11:09:20.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.203+0000 7f87a7fff640 1 -- 172.21.15.78:0/4087892464 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f87b0005020 con 0x7f87c01033e0 2024-03-16T11:09:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.204+0000 7f87a7fff640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f87a80a2800 0x7f87a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.204+0000 7f87bed9d640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f87a80a2800 0x7f87a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.204+0000 7f87a7fff640 1 -- 172.21.15.78:0/4087892464 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f87b00cf120 con 0x7f87c01033e0 2024-03-16T11:09:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.204+0000 7f87bed9d640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f87a80a2800 0x7f87a80a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f87c0102880 tx=0x7f87b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:20.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.208+0000 7f87a7fff640 1 -- 172.21.15.78:0/4087892464 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f87b009c1c0 con 0x7f87c01033e0 2024-03-16T11:09:20.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:20 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:09:20.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:20 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:09:20.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:20 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:09:20.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:20 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/302250303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:09:20.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:20 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2122992392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:09:20.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:20 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4029624947' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T11:09:20.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.389+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87c00623e0 con 0x7f87c01033e0 2024-03-16T11:09:20.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.391+0000 7f87a7fff640 1 -- 172.21.15.78:0/4087892464 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f87b009bac0 con 0x7f87c01033e0 2024-03-16T11:09:20.391 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:09:20.392 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:09:20.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f87a80a2800 msgr2=0x7f87a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f87a80a2800 0x7f87a80a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f87c0102880 tx=0x7f87b8005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c01033e0 msgr2=0x7f87c019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c01033e0 0x7f87c019efe0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f87b000eb10 tx=0x7f87b000c2d0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 shutdown_connections 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c01033e0 0x7f87c019efe0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f87a80a2800 0x7f87a80a4cc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c0102a20 0x7f87c0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 --2- 172.21.15.78:0/4087892464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c0101830 0x7f87c0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 >> 172.21.15.78:0/4087892464 conn(0x7f87c00fcd80 msgr2=0x7f87c0105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 shutdown_connections 2024-03-16T11:09:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.393+0000 7f87c50b2640 1 -- 172.21.15.78:0/4087892464 wait complete. 2024-03-16T11:09:20.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:59:26.228697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:03:58.210755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:03:58.851667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:09:20.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:09:20.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:20 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:09:20.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:20 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:09:20.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:20 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:09:20.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:20 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/302250303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:09:20.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:20 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2122992392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:09:20.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:20 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4029624947' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 -- 172.21.15.78:0/1134762821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 msgr2=0x7fe0fc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 --2- 172.21.15.78:0/1134762821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc108b40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009ca0 tx=0x7fe0f402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 -- 172.21.15.78:0/1134762821 shutdown_connections 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 --2- 172.21.15.78:0/1134762821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc108b40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 --2- 172.21.15.78:0/1134762821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0fc1018f0 0x7fe0fc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 --2- 172.21.15.78:0/1134762821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc100700 0x7fe0fc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 -- 172.21.15.78:0/1134762821 >> 172.21.15.78:0/1134762821 conn(0x7fe0fc0fbc90 msgr2=0x7fe0fc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 -- 172.21.15.78:0/1134762821 shutdown_connections 2024-03-16T11:09:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.509+0000 7fe103989640 1 -- 172.21.15.78:0/1134762821 wait complete. 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 Processor -- start 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 -- start start 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0fc100700 0x7fe0fc1968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc1018f0 0x7fe0fc196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc10c230 con 0x7fe0fc1022b0 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc10c0b0 con 0x7fe0fc1018f0 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe103989640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc10c3b0 con 0x7fe0fc100700 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe101f26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe101f26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52456/0 (socket says 172.21.15.78:52456) 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe101f26640 1 -- 172.21.15.78:0/3147006674 learned_addr learned my addr 172.21.15.78:0/3147006674 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe100f24640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc1018f0 0x7fe0fc196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.510+0000 7fe101725640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0fc100700 0x7fe0fc1968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe101f26640 1 -- 172.21.15.78:0/3147006674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0fc100700 msgr2=0x7fe0fc1968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe101f26640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0fc100700 0x7fe0fc1968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe101f26640 1 -- 172.21.15.78:0/3147006674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc1018f0 msgr2=0x7fe0fc196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe101f26640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc1018f0 0x7fe0fc196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe101f26640 1 -- 172.21.15.78:0/3147006674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0fc19e4c0 con 0x7fe0fc1022b0 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe101f26640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc19de50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe0f402c700 tx=0x7fe0f40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe0f27fc640 1 -- 172.21.15.78:0/3147006674 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4041020 con 0x7fe0fc1022b0 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe0f27fc640 1 -- 172.21.15.78:0/3147006674 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe0f4003e20 con 0x7fe0fc1022b0 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0fc19e750 con 0x7fe0fc1022b0 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe0f27fc640 1 -- 172.21.15.78:0/3147006674 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4039040 con 0x7fe0fc1022b0 2024-03-16T11:09:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.511+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0fc19ec40 con 0x7fe0fc1022b0 2024-03-16T11:09:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.512+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0fc107a20 con 0x7fe0fc1022b0 2024-03-16T11:09:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.513+0000 7fe0f27fc640 1 -- 172.21.15.78:0/3147006674 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe0f4044020 con 0x7fe0fc1022b0 2024-03-16T11:09:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.514+0000 7fe0f27fc640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe0e80a2800 0x7fe0e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.514+0000 7fe101725640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe0e80a2800 0x7fe0e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.514+0000 7fe0f27fc640 1 -- 172.21.15.78:0/3147006674 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe0f40f0c60 con 0x7fe0fc1022b0 2024-03-16T11:09:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.514+0000 7fe101725640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe0e80a2800 0x7fe0e80a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc101750 tx=0x7fe0f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:20.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.518+0000 7fe0f27fc640 1 -- 172.21.15.78:0/3147006674 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe0f404b020 con 0x7fe0fc1022b0 2024-03-16T11:09:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:20 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:09:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:20 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:09:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:20 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' 2024-03-16T11:09:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:20 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/302250303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:09:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:20 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2122992392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:09:20.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:20 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4029624947' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T11:09:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.692+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe0fc0623e0 con 0x7fe0fc1022b0 2024-03-16T11:09:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.692+0000 7fe0f27fc640 1 -- 172.21.15.78:0/3147006674 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fe0f40bdd00 con 0x7fe0fc1022b0 2024-03-16T11:09:20.693 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe0e80a2800 msgr2=0x7fe0e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe0e80a2800 0x7fe0e80a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc101750 tx=0x7fe0f8005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 msgr2=0x7fe0fc19de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc19de50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe0f402c700 tx=0x7fe0f40328a0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 shutdown_connections 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0fc1022b0 0x7fe0fc19de50 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0fc1018f0 0x7fe0fc196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fe0e80a2800 0x7fe0e80a4cc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 --2- 172.21.15.78:0/3147006674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0fc100700 0x7fe0fc1968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 >> 172.21.15.78:0/3147006674 conn(0x7fe0fc0fbc90 msgr2=0x7fe0fc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 shutdown_connections 2024-03-16T11:09:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.694+0000 7fe103989640 1 -- 172.21.15.78:0/3147006674 wait complete. 2024-03-16T11:09:20.706 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 -- 172.21.15.78:0/4153759129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 msgr2=0x7fb854108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 --2- 172.21.15.78:0/4153759129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb854108b40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009ca0 tx=0x7fb84c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 -- 172.21.15.78:0/4153759129 shutdown_connections 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 --2- 172.21.15.78:0/4153759129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb854108b40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 --2- 172.21.15.78:0/4153759129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8541018f0 0x7fb854101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 --2- 172.21.15.78:0/4153759129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb854100700 0x7fb854100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 -- 172.21.15.78:0/4153759129 >> 172.21.15.78:0/4153759129 conn(0x7fb8540fbc90 msgr2=0x7fb8540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 -- 172.21.15.78:0/4153759129 shutdown_connections 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.810+0000 7fb859177640 1 -- 172.21.15.78:0/4153759129 wait complete. 2024-03-16T11:09:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 Processor -- start 2024-03-16T11:09:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 -- start start 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100700 0x7fb8541979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8541018f0 0x7fb854197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb85419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb85410c230 con 0x7fb8541022b0 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb85410c0b0 con 0x7fb854100700 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb859177640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb85410c3b0 con 0x7fb8541018f0 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb85419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb84a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8541018f0 0x7fb854197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb85419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52474/0 (socket says 172.21.15.78:52474) 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 -- 172.21.15.78:0/3697515059 learned_addr learned my addr 172.21.15.78:0/3697515059 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb852d9d640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100700 0x7fb8541979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 -- 172.21.15.78:0/3697515059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8541018f0 msgr2=0x7fb854197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8541018f0 0x7fb854197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 -- 172.21.15.78:0/3697515059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100700 msgr2=0x7fb8541979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100700 0x7fb8541979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.811+0000 7fb85359e640 1 -- 172.21.15.78:0/3697515059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85419f610 con 0x7fb8541022b0 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.812+0000 7fb85359e640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb85419efa0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb84c02c700 tx=0x7fb84c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.812+0000 7fb850d99640 1 -- 172.21.15.78:0/3697515059 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c041020 con 0x7fb8541022b0 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.812+0000 7fb850d99640 1 -- 172.21.15.78:0/3697515059 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb84c003e20 con 0x7fb8541022b0 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.812+0000 7fb850d99640 1 -- 172.21.15.78:0/3697515059 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c039040 con 0x7fb8541022b0 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.812+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb85419f8a0 con 0x7fb8541022b0 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.812+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb85419fd40 con 0x7fb8541022b0 2024-03-16T11:09:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.813+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb85419ffa0 con 0x7fb8541022b0 2024-03-16T11:09:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.814+0000 7fb850d99640 1 -- 172.21.15.78:0/3697515059 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb84c044020 con 0x7fb8541022b0 2024-03-16T11:09:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.814+0000 7fb850d99640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8400a2800 0x7fb8400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.815+0000 7fb852d9d640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8400a2800 0x7fb8400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.815+0000 7fb850d99640 1 -- 172.21.15.78:0/3697515059 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb84c0f0c60 con 0x7fb8541022b0 2024-03-16T11:09:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.815+0000 7fb852d9d640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8400a2800 0x7fb8400a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb854101750 tx=0x7fb844005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:20.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.819+0000 7fb850d99640 1 -- 172.21.15.78:0/3697515059 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb84c04b020 con 0x7fb8541022b0 2024-03-16T11:09:20.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.993+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb8541a0170 con 0x7fb8541022b0 2024-03-16T11:09:20.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.993+0000 7fb850d99640 1 -- 172.21.15.78:0/3697515059 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fb84c0bdd00 con 0x7fb8541022b0 2024-03-16T11:09:20.994 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8400a2800 msgr2=0x7fb8400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8400a2800 0x7fb8400a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb854101750 tx=0x7fb844005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 msgr2=0x7fb85419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb85419efa0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb84c02c700 tx=0x7fb84c0328a0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 shutdown_connections 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8541022b0 0x7fb85419efa0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fb8400a2800 0x7fb8400a4cc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8541018f0 0x7fb854197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 --2- 172.21.15.78:0/3697515059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100700 0x7fb8541979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 >> 172.21.15.78:0/3697515059 conn(0x7fb8540fbc90 msgr2=0x7fb854069d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 shutdown_connections 2024-03-16T11:09:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:20.995+0000 7fb859177640 1 -- 172.21.15.78:0/3697515059 wait complete. 2024-03-16T11:09:21.007 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:09:21.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 -- 172.21.15.78:0/284770442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 msgr2=0x7f0190108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 --2- 172.21.15.78:0/284770442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f0190108b30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0188009ca0 tx=0x7f018802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 -- 172.21.15.78:0/284770442 shutdown_connections 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 --2- 172.21.15.78:0/284770442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f0190108b30 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 --2- 172.21.15.78:0/284770442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01901018e0 0x7f0190101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 --2- 172.21.15.78:0/284770442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01901006f0 0x7f0190100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 -- 172.21.15.78:0/284770442 >> 172.21.15.78:0/284770442 conn(0x7f01900fbc80 msgr2=0x7f01900fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 -- 172.21.15.78:0/284770442 shutdown_connections 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 -- 172.21.15.78:0/284770442 wait complete. 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 Processor -- start 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.112+0000 7f0196a9a640 1 -- start start 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0196a9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01901006f0 0x7f019006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0196a9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01901018e0 0x7f019019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0196a9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f019019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0196a9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f019010c200 con 0x7f01901022a0 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0196a9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f019010c080 con 0x7f01901018e0 2024-03-16T11:09:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0196a9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f019010c380 con 0x7f01901006f0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f019019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f019019ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52492/0 (socket says 172.21.15.78:52492) 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 -- 172.21.15.78:0/1720146027 learned_addr learned my addr 172.21.15.78:0/1720146027 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f018ffff640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01901018e0 0x7f019019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0194836640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01901006f0 0x7f019006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 -- 172.21.15.78:0/1720146027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01901006f0 msgr2=0x7f019006a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01901006f0 0x7f019006a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 -- 172.21.15.78:0/1720146027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01901018e0 msgr2=0x7f019019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01901018e0 0x7f019019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 -- 172.21.15.78:0/1720146027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019019f650 con 0x7f01901022a0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0195037640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f019019ed40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f018802c700 tx=0x7f01880328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f018dffb640 1 -- 172.21.15.78:0/1720146027 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188041020 con 0x7f01901022a0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f019019f880 con 0x7f01901022a0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f018dffb640 1 -- 172.21.15.78:0/1720146027 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0188003e20 con 0x7f01901022a0 2024-03-16T11:09:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.113+0000 7f018dffb640 1 -- 172.21.15.78:0/1720146027 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188039040 con 0x7f01901022a0 2024-03-16T11:09:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.114+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f019019fd70 con 0x7f01901022a0 2024-03-16T11:09:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.115+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0190107b30 con 0x7f01901022a0 2024-03-16T11:09:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.116+0000 7f018dffb640 1 -- 172.21.15.78:0/1720146027 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0188044020 con 0x7f01901022a0 2024-03-16T11:09:21.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.116+0000 7f018dffb640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01800a2800 0x7f01800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.116+0000 7f0194836640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01800a2800 0x7f01800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.116+0000 7f018dffb640 1 -- 172.21.15.78:0/1720146027 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f01880f0bf0 con 0x7f01901022a0 2024-03-16T11:09:21.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.117+0000 7f0194836640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01800a2800 0x7f01800a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f01840095f0 tx=0x7f0184005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:21.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.121+0000 7f018dffb640 1 -- 172.21.15.78:0/1720146027 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f018804b020 con 0x7f01901022a0 2024-03-16T11:09:21.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.295+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f01900623e0 con 0x7f01901022a0 2024-03-16T11:09:21.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.296+0000 7f018dffb640 1 -- 172.21.15.78:0/1720146027 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f01880bdc90 con 0x7f01901022a0 2024-03-16T11:09:21.296 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01800a2800 msgr2=0x7f01800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01800a2800 0x7f01800a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f01840095f0 tx=0x7f0184005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 msgr2=0x7f019019ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f019019ed40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f018802c700 tx=0x7f01880328a0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 shutdown_connections 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01901022a0 0x7f019019ed40 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01901018e0 0x7f019019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f01800a2800 0x7f01800a4cc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 --2- 172.21.15.78:0/1720146027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01901006f0 0x7f019006a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 >> 172.21.15.78:0/1720146027 conn(0x7f01900fbc80 msgr2=0x7f0190104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:21.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 shutdown_connections 2024-03-16T11:09:21.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.298+0000 7f0196a9a640 1 -- 172.21.15.78:0/1720146027 wait complete. 2024-03-16T11:09:21.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:21 smithi078 ceph-mon[32235]: pgmap v449: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:21.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:21 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/4087892464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:21.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:21 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3147006674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:09:21.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:21 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3697515059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:09:21.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T11:09:21.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 -- 172.21.15.78:0/119901260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 msgr2=0x7fcfd8101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 --2- 172.21.15.78:0/119901260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8101d60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009900 tx=0x7fcfc002bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 -- 172.21.15.78:0/119901260 shutdown_connections 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 --2- 172.21.15.78:0/119901260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd81022a0 0x7fcfd8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 --2- 172.21.15.78:0/119901260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8101d60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 --2- 172.21.15.78:0/119901260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81006f0 0x7fcfd8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 -- 172.21.15.78:0/119901260 >> 172.21.15.78:0/119901260 conn(0x7fcfd80fbc80 msgr2=0x7fcfd80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 -- 172.21.15.78:0/119901260 shutdown_connections 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.412+0000 7fcfdd710640 1 -- 172.21.15.78:0/119901260 wait complete. 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 Processor -- start 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 -- start start 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81006f0 0x7fcfd8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd81022a0 0x7fcfd819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd810c200 con 0x7fcfd81018e0 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd810c080 con 0x7fcfd81006f0 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfdd710640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd810c380 con 0x7fcfd81022a0 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfcffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfcffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52512/0 (socket says 172.21.15.78:52512) 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfcffff640 1 -- 172.21.15.78:0/844911447 learned_addr learned my addr 172.21.15.78:0/844911447 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfd6ffd640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81006f0 0x7fcfd8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.413+0000 7fcfd77fe640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd81022a0 0x7fcfd819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfcffff640 1 -- 172.21.15.78:0/844911447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd81022a0 msgr2=0x7fcfd819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfcffff640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd81022a0 0x7fcfd819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfcffff640 1 -- 172.21.15.78:0/844911447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81006f0 msgr2=0x7fcfd8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfcffff640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81006f0 0x7fcfd8197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfcffff640 1 -- 172.21.15.78:0/844911447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd819f660 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfcffff640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8197f80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0006010 tx=0x7fcfc0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfd4ff9640 1 -- 172.21.15.78:0/844911447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0043020 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfd4ff9640 1 -- 172.21.15.78:0/844911447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfc00043d0 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd819f8f0 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfd4ff9640 1 -- 172.21.15.78:0/844911447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0003b80 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.414+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd819fe90 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.415+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd81a0060 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.416+0000 7fcfd4ff9640 1 -- 172.21.15.78:0/844911447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fcfc0046020 con 0x7fcfd81018e0 2024-03-16T11:09:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.416+0000 7fcfd4ff9640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfc40a2880 0x7fcfc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:21.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.416+0000 7fcfd6ffd640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfc40a2880 0x7fcfc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:21.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.417+0000 7fcfd4ff9640 1 -- 172.21.15.78:0/844911447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcfc00f0bd0 con 0x7fcfd81018e0 2024-03-16T11:09:21.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.417+0000 7fcfd6ffd640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfc40a2880 0x7fcfc40a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80095f0 tx=0x7fcfc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:21.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.421+0000 7fcfd4ff9640 1 -- 172.21.15.78:0/844911447 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcfc004d020 con 0x7fcfd81018e0 2024-03-16T11:09:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:21 smithi098 ceph-mon[35848]: pgmap v449: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:21 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/4087892464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:21.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:21 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3147006674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:09:21.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:21 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3697515059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:09:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:21 smithi153 ceph-mon[35521]: pgmap v449: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:09:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:21 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/4087892464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:21 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3147006674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:09:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:21 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3697515059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:09:21.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.609+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcfd81a03f0 con 0x7fcfd81018e0 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.609+0000 7fcfd4ff9640 1 -- 172.21.15.78:0/844911447 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+591 (secure 0 0 0) 0x7fcfc00bdc70 con 0x7fcfd81018e0 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/524003541 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2003716687 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3269664941 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2003716687 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1201370297 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2103714756 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1410130144 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3269664941 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2522710278 2024-03-17T10:56:34.960962+0000 2024-03-16T11:09:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4255056165 2024-03-17T10:57:01.884008+0000 2024-03-16T11:09:21.611 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-03-16T11:09:21.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfc40a2880 msgr2=0x7fcfc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfc40a2880 0x7fcfc40a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80095f0 tx=0x7fcfc8005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 msgr2=0x7fcfd8197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8197f80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0006010 tx=0x7fcfc0005e40 comp rx=0 tx=0).stop 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 shutdown_connections 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fcfc40a2880 0x7fcfc40a4d40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd81022a0 0x7fcfd819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfd81018e0 0x7fcfd8197f80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 --2- 172.21.15.78:0/844911447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81006f0 0x7fcfd8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 >> 172.21.15.78:0/844911447 conn(0x7fcfd80fbc80 msgr2=0x7fcfd8069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 shutdown_connections 2024-03-16T11:09:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:21.611+0000 7fcfdd710640 1 -- 172.21.15.78:0/844911447 wait complete. 2024-03-16T11:09:21.623 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-03-16T11:09:21.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:09:21.623 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-16T11:09:21.650 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T11:09:21.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:ceph-fuse[84740]: fuse finished with error 0 and tester_r 0 2024-03-16T11:09:22.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:22 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1720146027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:09:22.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:22 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/844911447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T11:09:22.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:22 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1720146027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:09:22.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:22 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/844911447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T11:09:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:22 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1720146027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:09:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:22 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/844911447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T11:09:23.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:23 smithi098 ceph-mon[35848]: pgmap v450: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:23 smithi153 ceph-mon[35521]: pgmap v450: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:23.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:23 smithi078 ceph-mon[32235]: pgmap v450: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:09:25.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:25 smithi098 ceph-mon[35848]: pgmap v451: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:09:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:25 smithi153 ceph-mon[35521]: pgmap v451: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:09:25.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:25 smithi078 ceph-mon[32235]: pgmap v451: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T11:09:27.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:27 smithi098 ceph-mon[35848]: pgmap v452: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:09:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:27 smithi153 ceph-mon[35521]: pgmap v452: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:09:27.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:27 smithi078 ceph-mon[32235]: pgmap v452: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:09:27.752 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T11:09:27.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:09:27.753 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-16T11:09:27.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:09:27.784 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-16T11:09:27.842 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-16T11:09:27.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:09:27.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T11:09:27.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:09:27.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-16T11:09:28.026 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-16T11:09:28.038 INFO:tasks.cephadm:Teardown begin 2024-03-16T11:09:28.038 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-16T11:09:28.038 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:09:28.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- 172.21.15.78:0/1630564699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801018f0 msgr2=0x7f7f80101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 --2- 172.21.15.78:0/1630564699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801018f0 0x7f7f80101d70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009900 tx=0x7f7f7002bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- 172.21.15.78:0/1630564699 shutdown_connections 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 --2- 172.21.15.78:0/1630564699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f801022b0 0x7f7f80108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 --2- 172.21.15.78:0/1630564699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801018f0 0x7f7f80101d70 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 --2- 172.21.15.78:0/1630564699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f80100700 0x7f7f80100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- 172.21.15.78:0/1630564699 >> 172.21.15.78:0/1630564699 conn(0x7f7f800fbc70 msgr2=0x7f7f800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- 172.21.15.78:0/1630564699 shutdown_connections 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- 172.21.15.78:0/1630564699 wait complete. 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 Processor -- start 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- start start 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f80100700 0x7f7f800694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f801018f0 0x7f7f8019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801022b0 0x7f7f8019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8010c230 con 0x7f7f801022b0 2024-03-16T11:09:28.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8010c0b0 con 0x7f7f80100700 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.154+0000 7f7f88636640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8010c3b0 con 0x7f7f801018f0 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801022b0 0x7f7f8019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801022b0 0x7f7f8019edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33354/0 (socket says 172.21.15.78:33354) 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 -- 172.21.15.78:0/2082448745 learned_addr learned my addr 172.21.15.78:0/2082448745 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f85bd1640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f801018f0 0x7f7f8019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f863d2640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f80100700 0x7f7f800694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 -- 172.21.15.78:0/2082448745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f801018f0 msgr2=0x7f7f8019c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f801018f0 0x7f7f8019c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 -- 172.21.15.78:0/2082448745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f80100700 msgr2=0x7f7f800694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f80100700 0x7f7f800694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 -- 172.21.15.78:0/2082448745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8019f6d0 con 0x7f7f801022b0 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f86bd3640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801022b0 0x7f7f8019edc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800eb10 tx=0x7f7f7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f777fe640 1 -- 172.21.15.78:0/2082448745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78004b20 con 0x7f7f801022b0 2024-03-16T11:09:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f777fe640 1 -- 172.21.15.78:0/2082448745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f78004cc0 con 0x7f7f801022b0 2024-03-16T11:09:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f777fe640 1 -- 172.21.15.78:0/2082448745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78010690 con 0x7f7f801022b0 2024-03-16T11:09:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8019f960 con 0x7f7f801022b0 2024-03-16T11:09:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.155+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f8019fe30 con 0x7f7f801022b0 2024-03-16T11:09:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.156+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f80107af0 con 0x7f7f801022b0 2024-03-16T11:09:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.157+0000 7f7f777fe640 1 -- 172.21.15.78:0/2082448745 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f7f78005000 con 0x7f7f801022b0 2024-03-16T11:09:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.158+0000 7f7f777fe640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7f6c0a2800 0x7f7f6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.158+0000 7f7f863d2640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7f6c0a2800 0x7f7f6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.158+0000 7f7f777fe640 1 -- 172.21.15.78:0/2082448745 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7f780cf1b0 con 0x7f7f801022b0 2024-03-16T11:09:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.158+0000 7f7f863d2640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7f6c0a2800 0x7f7f6c0a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7f80101750 tx=0x7f7f7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:28.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.161+0000 7f7f777fe640 1 -- 172.21.15.78:0/2082448745 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f78053740 con 0x7f7f801022b0 2024-03-16T11:09:28.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.381+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f8002d370 con 0x7f7f801022b0 2024-03-16T11:09:28.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.382+0000 7f7f777fe640 1 -- 172.21.15.78:0/2082448745 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7f7f7809c250 con 0x7f7f801022b0 2024-03-16T11:09:28.383 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:09:28.383 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:09:28.383 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7f6c0a2800 msgr2=0x7f7f6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7f6c0a2800 0x7f7f6c0a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7f80101750 tx=0x7f7f7c005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801022b0 msgr2=0x7f7f8019edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801022b0 0x7f7f8019edc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800eb10 tx=0x7f7f7800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 shutdown_connections 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f801022b0 0x7f7f8019edc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f7f6c0a2800 0x7f7f6c0a4cc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f801018f0 0x7f7f8019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 --2- 172.21.15.78:0/2082448745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f80100700 0x7f7f800694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 >> 172.21.15.78:0/2082448745 conn(0x7f7f800fbc70 msgr2=0x7f7f80104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 shutdown_connections 2024-03-16T11:09:28.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.384+0000 7f7f88636640 1 -- 172.21.15.78:0/2082448745 wait complete. 2024-03-16T11:09:28.396 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:09:28.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 -- 172.21.15.78:0/3783217261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 msgr2=0x7ff800109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 --2- 172.21.15.78:0/3783217261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff800109c40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009ca0 tx=0x7ff7fc02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 -- 172.21.15.78:0/3783217261 shutdown_connections 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 --2- 172.21.15.78:0/3783217261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff800109c40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 --2- 172.21.15.78:0/3783217261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8001029f0 0x7ff800102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 --2- 172.21.15.78:0/3783217261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff800101800 0x7ff800101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 -- 172.21.15.78:0/3783217261 >> 172.21.15.78:0/3783217261 conn(0x7ff8000fcd90 msgr2=0x7ff8000ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 -- 172.21.15.78:0/3783217261 shutdown_connections 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.498+0000 7ff808685640 1 -- 172.21.15.78:0/3783217261 wait complete. 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 Processor -- start 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 -- start start 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff800101800 0x7ff8001979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8001029f0 0x7ff800197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff80019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff80010d330 con 0x7ff8001033b0 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff80010d1b0 con 0x7ff8001029f0 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff80010d4b0 con 0x7ff800101800 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff80019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff80019ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33358/0 (socket says 172.21.15.78:33358) 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 -- 172.21.15.78:0/878953957 learned_addr learned my addr 172.21.15.78:0/878953957 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806421640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff800101800 0x7ff8001979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff805c20640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8001029f0 0x7ff800197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 -- 172.21.15.78:0/878953957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff800101800 msgr2=0x7ff8001979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff800101800 0x7ff8001979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 -- 172.21.15.78:0/878953957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8001029f0 msgr2=0x7ff800197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8001029f0 0x7ff800197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 -- 172.21.15.78:0/878953957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80019f5e0 con 0x7ff8001033b0 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff806c22640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff80019ef70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc02c700 tx=0x7ff7fc0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff7f77fe640 1 -- 172.21.15.78:0/878953957 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc041020 con 0x7ff8001033b0 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80019f870 con 0x7ff8001033b0 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff7f77fe640 1 -- 172.21.15.78:0/878953957 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7fc003e20 con 0x7ff8001033b0 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff7f77fe640 1 -- 172.21.15.78:0/878953957 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc039040 con 0x7ff8001033b0 2024-03-16T11:09:28.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.499+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80019fd60 con 0x7ff8001033b0 2024-03-16T11:09:28.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.500+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff800108bc0 con 0x7ff8001033b0 2024-03-16T11:09:28.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.501+0000 7ff7f77fe640 1 -- 172.21.15.78:0/878953957 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ff7fc044020 con 0x7ff8001033b0 2024-03-16T11:09:28.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.502+0000 7ff7f77fe640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.502+0000 7ff7f77fe640 1 -- 172.21.15.78:0/878953957 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff7fc0f0d00 con 0x7ff8001033b0 2024-03-16T11:09:28.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.502+0000 7ff806421640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.502+0000 7ff806421640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff800102850 tx=0x7ff7f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:28.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.506+0000 7ff7f77fe640 1 -- 172.21.15.78:0/878953957 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7fc04b020 con 0x7ff8001033b0 2024-03-16T11:09:28.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.717+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8000623e0 con 0x7ff8001033b0 2024-03-16T11:09:28.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.718+0000 7ff7f77fe640 1 -- 172.21.15.78:0/878953957 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11869 (secure 0 0 0) 0x7ff7fc0bdda0 con 0x7ff8001033b0 2024-03-16T11:09:28.719 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:09:28.719 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1233514200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1233514200},{"type":"v1","addr":"172.21.15.78:6835","nonce":1233514200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4216287385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4216287385},{"type":"v1","addr":"172.21.15.78:6837","nonce":4216287385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/3781553715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":3781553715},{"type":"v1","addr":"172.21.15.78:6839","nonce":3781553715}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/627084813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":627084813},{"type":"v1","addr":"172.21.15.78:6841","nonce":627084813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/3200006820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3200006820},{"type":"v1","addr":"172.21.15.98:6833","nonce":3200006820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3909734389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3909734389},{"type":"v1","addr":"172.21.15.153:6833","nonce":3909734389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:03:59.617240+0000","modified":"2024-03-16T11:05:54.020975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24505,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6835/1190197046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1190197046},{"type":"v1","addr":"172.21.15.98:6835","nonce":1190197046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1109965992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1109965992},{"type":"v1","addr":"172.21.15.153:6835","nonce":1109965992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3505861862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3505861862},{"type":"v1","addr":"172.21.15.153:6837","nonce":3505861862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6837/1752557490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1752557490},{"type":"v1","addr":"172.21.15.98:6837","nonce":1752557490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.153:6839/1504943793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1504943793},{"type":"v1","addr":"172.21.15.153:6839","nonce":1504943793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6839/4092843053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4092843053},{"type":"v1","addr":"172.21.15.98:6839","nonce":4092843053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24488,24505]},"id":1}]} 2024-03-16T11:09:28.719 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff7ec0a2800 msgr2=0x7ff7ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff800102850 tx=0x7ff7f8005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 msgr2=0x7ff80019ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff80019ef70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc02c700 tx=0x7ff7fc0328a0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 shutdown_connections 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8001033b0 0x7ff80019ef70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8001029f0 0x7ff800197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7ff7ec0a2800 0x7ff7ec0a4cc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 --2- 172.21.15.78:0/878953957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff800101800 0x7ff8001979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 >> 172.21.15.78:0/878953957 conn(0x7ff8000fcd90 msgr2=0x7ff800105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 shutdown_connections 2024-03-16T11:09:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.720+0000 7ff808685640 1 -- 172.21.15.78:0/878953957 wait complete. 2024-03-16T11:09:28.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:09:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.833+0000 7f978e505640 1 -- 172.21.15.78:0/237667780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9788100700 msgr2=0x7f9788100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.833+0000 7f978e505640 1 --2- 172.21.15.78:0/237667780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9788100700 0x7f9788100b20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9778009900 tx=0x7f977802bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.833+0000 7f978e505640 1 -- 172.21.15.78:0/237667780 shutdown_connections 2024-03-16T11:09:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.833+0000 7f978e505640 1 --2- 172.21.15.78:0/237667780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97881022b0 0x7f9788108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.833+0000 7f978e505640 1 --2- 172.21.15.78:0/237667780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97881018f0 0x7f9788101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.833+0000 7f978e505640 1 --2- 172.21.15.78:0/237667780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9788100700 0x7f9788100b20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.833+0000 7f978e505640 1 -- 172.21.15.78:0/237667780 >> 172.21.15.78:0/237667780 conn(0x7f97880fbc70 msgr2=0x7f97880fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 -- 172.21.15.78:0/237667780 shutdown_connections 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 -- 172.21.15.78:0/237667780 wait complete. 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 Processor -- start 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 -- start start 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788100700 0x7f9788197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97881018f0 0x7f9788197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97881022b0 0x7f978819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f978810c220 con 0x7f97881018f0 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f978810c0a0 con 0x7f97881022b0 2024-03-16T11:09:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f978e505640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f978810c3a0 con 0x7f9788100700 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.834+0000 7f97877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97881018f0 0x7f9788197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97881018f0 0x7f9788197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33366/0 (socket says 172.21.15.78:33366) 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 -- 172.21.15.78:0/807195682 learned_addr learned my addr 172.21.15.78:0/807195682 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f9787fff640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788100700 0x7f9788197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f978caa2640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97881022b0 0x7f978819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 -- 172.21.15.78:0/807195682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788100700 msgr2=0x7f9788197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788100700 0x7f9788197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 -- 172.21.15.78:0/807195682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97881022b0 msgr2=0x7f978819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97881022b0 0x7f978819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 -- 172.21.15.78:0/807195682 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978819f6c0 con 0x7f97881018f0 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97877fe640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97881018f0 0x7f9788197fe0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f977c00e9f0 tx=0x7f977c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97857fa640 1 -- 172.21.15.78:0/807195682 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c00cd80 con 0x7f97881018f0 2024-03-16T11:09:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97857fa640 1 -- 172.21.15.78:0/807195682 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f977c0054d0 con 0x7f97881018f0 2024-03-16T11:09:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978819f9b0 con 0x7f97881018f0 2024-03-16T11:09:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f97857fa640 1 -- 172.21.15.78:0/807195682 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c010490 con 0x7f97881018f0 2024-03-16T11:09:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.835+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978819ff00 con 0x7f97881018f0 2024-03-16T11:09:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.836+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97881a0190 con 0x7f97881018f0 2024-03-16T11:09:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.837+0000 7f97857fa640 1 -- 172.21.15.78:0/807195682 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f977c005000 con 0x7f97881018f0 2024-03-16T11:09:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.838+0000 7f97857fa640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97700a2880 0x7f97700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.838+0000 7f9787fff640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97700a2880 0x7f97700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.838+0000 7f97857fa640 1 -- 172.21.15.78:0/807195682 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f977c0cef10 con 0x7f97881018f0 2024-03-16T11:09:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.838+0000 7f9787fff640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97700a2880 0x7f97700a4d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9778005ab0 tx=0x7f9778005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:28.841+0000 7f97857fa640 1 -- 172.21.15.78:0/807195682 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f977c053740 con 0x7f97881018f0 2024-03-16T11:09:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.018+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97881a0360 con 0x7f97881018f0 2024-03-16T11:09:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.020+0000 7f97857fa640 1 -- 172.21.15.78:0/807195682 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22614 (secure 0 0 0) 0x7f977c0052f0 con 0x7f97881018f0 2024-03-16T11:09:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-16T11:09:29.021 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":90,"fsid":"a1941e5e-e383-11ee-95c9-87774f69a715","created":"2024-03-16T10:56:07.773806+0000","modified":"2024-03-16T11:05:03.097866+0000","last_up_change":"2024-03-16T11:03:09.103120+0000","last_in_change":"2024-03-16T11:02:49.304278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:59:26.228697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:03:58.210755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:03:58.851667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"543437a6-61d5-4969-be4e-7a20f1edc236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6803","nonce":467332989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6805","nonce":467332989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6809","nonce":467332989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":467332989},{"type":"v1","addr":"172.21.15.78:6807","nonce":467332989}]},"public_addr":"172.21.15.78:6803/467332989","cluster_addr":"172.21.15.78:6805/467332989","heartbeat_back_addr":"172.21.15.78:6809/467332989","heartbeat_front_addr":"172.21.15.78:6807/467332989","state":["exists","up"]},{"osd":1,"uuid":"2c57f283-922d-4ef4-a00d-ce2f62a7e19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6801","nonce":64709971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6803","nonce":64709971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6807","nonce":64709971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":64709971},{"type":"v1","addr":"172.21.15.98:6805","nonce":64709971}]},"public_addr":"172.21.15.98:6801/64709971","cluster_addr":"172.21.15.98:6803/64709971","heartbeat_back_addr":"172.21.15.98:6807/64709971","heartbeat_front_addr":"172.21.15.98:6805/64709971","state":["exists","up"]},{"osd":2,"uuid":"4559999e-89a9-433d-b3b8-bdec51aebeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6801","nonce":2715466096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6803","nonce":2715466096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6807","nonce":2715466096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2715466096},{"type":"v1","addr":"172.21.15.153:6805","nonce":2715466096}]},"public_addr":"172.21.15.153:6801/2715466096","cluster_addr":"172.21.15.153:6803/2715466096","heartbeat_back_addr":"172.21.15.153:6807/2715466096","heartbeat_front_addr":"172.21.15.153:6805/2715466096","state":["exists","up"]},{"osd":3,"uuid":"e3a68ec8-b4f9-42fb-9f85-e683ef538fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6811","nonce":3489770309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6813","nonce":3489770309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6817","nonce":3489770309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3489770309},{"type":"v1","addr":"172.21.15.78:6815","nonce":3489770309}]},"public_addr":"172.21.15.78:6811/3489770309","cluster_addr":"172.21.15.78:6813/3489770309","heartbeat_back_addr":"172.21.15.78:6817/3489770309","heartbeat_front_addr":"172.21.15.78:6815/3489770309","state":["exists","up"]},{"osd":4,"uuid":"9d39555d-45cf-4507-80f8-17103f55bf4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6809","nonce":2809778583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6811","nonce":2809778583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6815","nonce":2809778583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2809778583},{"type":"v1","addr":"172.21.15.98:6813","nonce":2809778583}]},"public_addr":"172.21.15.98:6809/2809778583","cluster_addr":"172.21.15.98:6811/2809778583","heartbeat_back_addr":"172.21.15.98:6815/2809778583","heartbeat_front_addr":"172.21.15.98:6813/2809778583","state":["exists","up"]},{"osd":5,"uuid":"9e25d0ef-c9d0-4db6-b6b3-f1d00089d31f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6809","nonce":1494126150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6811","nonce":1494126150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6815","nonce":1494126150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1494126150},{"type":"v1","addr":"172.21.15.153:6813","nonce":1494126150}]},"public_addr":"172.21.15.153:6809/1494126150","cluster_addr":"172.21.15.153:6811/1494126150","heartbeat_back_addr":"172.21.15.153:6815/1494126150","heartbeat_front_addr":"172.21.15.153:6813/1494126150","state":["exists","up"]},{"osd":6,"uuid":"c4ef5595-a5ca-4f0e-82a7-f8322daab5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6819","nonce":1730932549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6821","nonce":1730932549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6825","nonce":1730932549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1730932549},{"type":"v1","addr":"172.21.15.78:6823","nonce":1730932549}]},"public_addr":"172.21.15.78:6819/1730932549","cluster_addr":"172.21.15.78:6821/1730932549","heartbeat_back_addr":"172.21.15.78:6825/1730932549","heartbeat_front_addr":"172.21.15.78:6823/1730932549","state":["exists","up"]},{"osd":7,"uuid":"db466cbf-77b8-4cc4-95ca-a49e51ba27e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6817","nonce":999107021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6819","nonce":999107021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6823","nonce":999107021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":999107021},{"type":"v1","addr":"172.21.15.98:6821","nonce":999107021}]},"public_addr":"172.21.15.98:6817/999107021","cluster_addr":"172.21.15.98:6819/999107021","heartbeat_back_addr":"172.21.15.98:6823/999107021","heartbeat_front_addr":"172.21.15.98:6821/999107021","state":["exists","up"]},{"osd":8,"uuid":"e063f0ca-e609-4d53-8a22-f6ce8f3dae43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6817","nonce":208983464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6819","nonce":208983464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6823","nonce":208983464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":208983464},{"type":"v1","addr":"172.21.15.153:6821","nonce":208983464}]},"public_addr":"172.21.15.153:6817/208983464","cluster_addr":"172.21.15.153:6819/208983464","heartbeat_back_addr":"172.21.15.153:6823/208983464","heartbeat_front_addr":"172.21.15.153:6821/208983464","state":["exists","up"]},{"osd":9,"uuid":"66171eea-9d12-44e4-b449-b83fa2a1d52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6827","nonce":2205607708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6829","nonce":2205607708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6833","nonce":2205607708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2205607708},{"type":"v1","addr":"172.21.15.78:6831","nonce":2205607708}]},"public_addr":"172.21.15.78:6827/2205607708","cluster_addr":"172.21.15.78:6829/2205607708","heartbeat_back_addr":"172.21.15.78:6833/2205607708","heartbeat_front_addr":"172.21.15.78:6831/2205607708","state":["exists","up"]},{"osd":10,"uuid":"72e85ff6-49af-4fc8-8114-9f33496e912b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6825","nonce":652996874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6827","nonce":652996874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6831","nonce":652996874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":652996874},{"type":"v1","addr":"172.21.15.98:6829","nonce":652996874}]},"public_addr":"172.21.15.98:6825/652996874","cluster_addr":"172.21.15.98:6827/652996874","heartbeat_back_addr":"172.21.15.98:6831/652996874","heartbeat_front_addr":"172.21.15.98:6829/652996874","state":["exists","up"]},{"osd":11,"uuid":"e383aa05-dedb-4d76-9e7a-0411d04ef5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6825","nonce":1068297198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6827","nonce":1068297198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6831","nonce":1068297198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1068297198},{"type":"v1","addr":"172.21.15.153:6829","nonce":1068297198}]},"public_addr":"172.21.15.153:6825/1068297198","cluster_addr":"172.21.15.153:6827/1068297198","heartbeat_back_addr":"172.21.15.153:6831/1068297198","heartbeat_front_addr":"172.21.15.153:6829/1068297198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:36.312801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:58:59.894340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:23.413457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:59:48.507091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:12.759735+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:00:36.663740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:01.533348+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:26.339691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:01:50.923328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:17.240844+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:02:42.340647+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:03:07.322676+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/524003541":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:6801/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6801/2003716687":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1201370297":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/2103714756":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/1410130144":"2024-03-17T10:57:01.884008+0000","172.21.15.78:6800/3269664941":"2024-03-17T10:57:01.884008+0000","172.21.15.78:0/2522710278":"2024-03-17T10:56:34.960962+0000","172.21.15.78:0/4255056165":"2024-03-17T10:57:01.884008+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97700a2880 msgr2=0x7f97700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97700a2880 0x7f97700a4d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9778005ab0 tx=0x7f9778005fd0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97881018f0 msgr2=0x7f9788197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97881018f0 0x7f9788197fe0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f977c00e9f0 tx=0x7f977c00edf0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 shutdown_connections 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97881022b0 0x7f978819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97881018f0 0x7f9788197fe0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f97700a2880 0x7f97700a4d40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 --2- 172.21.15.78:0/807195682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788100700 0x7f9788197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 >> 172.21.15.78:0/807195682 conn(0x7f97880fbc70 msgr2=0x7f97880689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 shutdown_connections 2024-03-16T11:09:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.022+0000 7f978e505640 1 -- 172.21.15.78:0/807195682 wait complete. 2024-03-16T11:09:29.035 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-16T11:09:29.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 -- 172.21.15.78:0/385125539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 msgr2=0x7fc4a4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 --2- 172.21.15.78:0/385125539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a4100b10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc494009900 tx=0x7fc49402bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:29.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 -- 172.21.15.78:0/385125539 shutdown_connections 2024-03-16T11:09:29.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 --2- 172.21.15.78:0/385125539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4a41022a0 0x7fc4a4108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 --2- 172.21.15.78:0/385125539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a41018e0 0x7fc4a4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 --2- 172.21.15.78:0/385125539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a4100b10 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 -- 172.21.15.78:0/385125539 >> 172.21.15.78:0/385125539 conn(0x7fc4a40fbc80 msgr2=0x7fc4a40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 -- 172.21.15.78:0/385125539 shutdown_connections 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 -- 172.21.15.78:0/385125539 wait complete. 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 Processor -- start 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.136+0000 7fc4a9885640 1 -- start start 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a406a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4a41018e0 0x7fc4a419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a41022a0 0x7fc4a419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a410c220 con 0x7fc4a41006f0 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a410c0a0 con 0x7fc4a41022a0 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a410c3a0 con 0x7fc4a41018e0 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a406a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a406a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33378/0 (socket says 172.21.15.78:33378) 2024-03-16T11:09:29.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 -- 172.21.15.78:0/1168077676 learned_addr learned my addr 172.21.15.78:0/1168077676 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:29.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a27fc640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4a41018e0 0x7fc4a419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a37fe640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a41022a0 0x7fc4a419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 -- 172.21.15.78:0/1168077676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4a41018e0 msgr2=0x7fc4a419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4a41018e0 0x7fc4a419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 -- 172.21.15.78:0/1168077676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a41022a0 msgr2=0x7fc4a419edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a41022a0 0x7fc4a419edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 -- 172.21.15.78:0/1168077676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a419f680 con 0x7fc4a41006f0 2024-03-16T11:09:29.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a2ffd640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a406a940 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc494000c00 tx=0x7fc494005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a8883640 1 -- 172.21.15.78:0/1168077676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494043020 con 0x7fc4a41006f0 2024-03-16T11:09:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a8883640 1 -- 172.21.15.78:0/1168077676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc494004350 con 0x7fc4a41006f0 2024-03-16T11:09:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a8883640 1 -- 172.21.15.78:0/1168077676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494003b80 con 0x7fc4a41006f0 2024-03-16T11:09:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a419f910 con 0x7fc4a41006f0 2024-03-16T11:09:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.137+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a419fe60 con 0x7fc4a41006f0 2024-03-16T11:09:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.138+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a4107b30 con 0x7fc4a41006f0 2024-03-16T11:09:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.139+0000 7fc4a8883640 1 -- 172.21.15.78:0/1168077676 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc494046020 con 0x7fc4a41006f0 2024-03-16T11:09:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.140+0000 7fc4a8883640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc48c0a2800 0x7fc48c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.140+0000 7fc4a27fc640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc48c0a2800 0x7fc48c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.140+0000 7fc4a8883640 1 -- 172.21.15.78:0/1168077676 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc4940f0f40 con 0x7fc4a41006f0 2024-03-16T11:09:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.140+0000 7fc4a27fc640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc48c0a2800 0x7fc48c0a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc4a406a6b0 tx=0x7fc498005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:29.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.143+0000 7fc4a8883640 1 -- 172.21.15.78:0/1168077676 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc49404d020 con 0x7fc4a41006f0 2024-03-16T11:09:29.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:29 smithi078 ceph-mon[32235]: pgmap v453: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:09:29.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:29 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/2082448745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:29 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/878953957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:29 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/807195682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.356+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc4a402d400 con 0x7fc4a41006f0 2024-03-16T11:09:29.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.358+0000 7fc4a8883640 1 -- 172.21.15.78:0/1168077676 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) ==== 161+0+0 (secure 0 0 0) 0x7fc4940bdfe0 con 0x7fc4a41006f0 2024-03-16T11:09:29.358 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-16T11:09:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.360+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc48c0a2800 msgr2=0x7fc48c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.360+0000 7fc4a9885640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc48c0a2800 0x7fc48c0a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc4a406a6b0 tx=0x7fc498005f90 comp rx=0 tx=0).stop 2024-03-16T11:09:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.360+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 msgr2=0x7fc4a406a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.360+0000 7fc4a9885640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a406a940 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc494000c00 tx=0x7fc494005dc0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 shutdown_connections 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a41022a0 0x7fc4a419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fc48c0a2800 0x7fc48c0a4cc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4a41018e0 0x7fc4a419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 --2- 172.21.15.78:0/1168077676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4a41006f0 0x7fc4a406a940 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 >> 172.21.15.78:0/1168077676 conn(0x7fc4a40fbc80 msgr2=0x7fc4a4104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 shutdown_connections 2024-03-16T11:09:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.361+0000 7fc4a9885640 1 -- 172.21.15.78:0/1168077676 wait complete. 2024-03-16T11:09:29.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-16T11:09:29.465 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: ignoring --setuser ceph since I am not root 2024-03-16T11:09:29.465 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: ignoring --setgroup ceph since I am not root 2024-03-16T11:09:29.465 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mds[67023]: -- 172.21.15.98:0/4142529475 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b81a69fa40 con 0x55b81a673000 2024-03-16T11:09:29.465 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: starting mds.b at 2024-03-16T11:09:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mon[35848]: pgmap v453: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:09:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/2082448745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/878953957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/807195682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h[66696]: ignoring --setuser ceph since I am not root 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h[66696]: ignoring --setgroup ceph since I am not root 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mds[66700]: -- 172.21.15.98:0/2142174975 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b97defda40 con 0x55b97ded1000 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h[66696]: starting mds.h at 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e[66385]: ignoring --setuser ceph since I am not root 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e[66385]: ignoring --setgroup ceph since I am not root 2024-03-16T11:09:29.466 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mds[66389]: -- 172.21.15.98:0/3981772204 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5578495761a0 con 0x55784954cc00 2024-03-16T11:09:29.467 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-mds[66389]: -- 172.21.15.98:0/3981772204 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557849579a40 con 0x55784954cc00 2024-03-16T11:09:29.467 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:09:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e[66385]: starting mds.e at 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 -- 172.21.15.78:0/141001622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101018f0 msgr2=0x7f2610101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 --2- 172.21.15.78:0/141001622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101018f0 0x7f2610101d70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009900 tx=0x7f25f802bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 -- 172.21.15.78:0/141001622 shutdown_connections 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 --2- 172.21.15.78:0/141001622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26101022b0 0x7f2610108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 --2- 172.21.15.78:0/141001622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101018f0 0x7f2610101d70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 --2- 172.21.15.78:0/141001622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2610100700 0x7f2610100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 -- 172.21.15.78:0/141001622 >> 172.21.15.78:0/141001622 conn(0x7f26100fbc90 msgr2=0x7f26100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 -- 172.21.15.78:0/141001622 shutdown_connections 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.479+0000 7f2615590640 1 -- 172.21.15.78:0/141001622 wait complete. 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 Processor -- start 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 -- start start 2024-03-16T11:09:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2610100700 0x7f2610197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26101018f0 0x7f2610197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101022b0 0x7f261019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d350 con 0x7f26101022b0 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d1d0 con 0x7f26101018f0 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2615590640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d4d0 con 0x7f2610100700 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101022b0 0x7f261019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101022b0 0x7f261019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33398/0 (socket says 172.21.15.78:33398) 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 -- 172.21.15.78:0/3242961532 learned_addr learned my addr 172.21.15.78:0/3242961532 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f2607fff640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26101018f0 0x7f2610197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260effd640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2610100700 0x7f2610197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 -- 172.21.15.78:0/3242961532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2610100700 msgr2=0x7f2610197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2610100700 0x7f2610197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 -- 172.21.15.78:0/3242961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26101018f0 msgr2=0x7f2610197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26101018f0 0x7f2610197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 -- 172.21.15.78:0/3242961532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261019f660 con 0x7f26101022b0 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260effd640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2610100700 0x7f2610197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260f7fe640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101022b0 0x7f261019eff0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f260800eb10 tx=0x7f260800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.480+0000 7f260cff9640 1 -- 172.21.15.78:0/3242961532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608004b20 con 0x7f26101022b0 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.481+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261019f950 con 0x7f26101022b0 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.481+0000 7f260cff9640 1 -- 172.21.15.78:0/3242961532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2608004cc0 con 0x7f26101022b0 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.481+0000 7f260cff9640 1 -- 172.21.15.78:0/3242961532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608010690 con 0x7f26101022b0 2024-03-16T11:09:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.481+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f261019fea0 con 0x7f26101022b0 2024-03-16T11:09:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.482+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2610107aa0 con 0x7f26101022b0 2024-03-16T11:09:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.483+0000 7f260cff9640 1 -- 172.21.15.78:0/3242961532 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2608005020 con 0x7f26101022b0 2024-03-16T11:09:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.483+0000 7f260cff9640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f25fc0a2800 0x7f25fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.483+0000 7f260effd640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f25fc0a2800 0x7f25fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.483+0000 7f260cff9640 1 -- 172.21.15.78:0/3242961532 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f26080cf220 con 0x7f26101022b0 2024-03-16T11:09:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.484+0000 7f260effd640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f25fc0a2800 0x7f25fc0a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2600005d00 tx=0x7f2600005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:29.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.487+0000 7f260cff9640 1 -- 172.21.15.78:0/3242961532 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f260809c1b0 con 0x7f26101022b0 2024-03-16T11:09:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-mon[35521]: pgmap v453: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:09:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/2082448745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/878953957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/807195682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:09:29.544 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c[66789]: ignoring --setuser ceph since I am not root 2024-03-16T11:09:29.544 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c[66789]: ignoring --setgroup ceph since I am not root 2024-03-16T11:09:29.544 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-mds[66793]: -- 172.21.15.153:0/484920024 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a4a5d7a40 con 0x563a4a5aa800 2024-03-16T11:09:29.544 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c[66789]: starting mds.c at 2024-03-16T11:09:29.544 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f[66481]: ignoring --setuser ceph since I am not root 2024-03-16T11:09:29.544 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f[66481]: ignoring --setgroup ceph since I am not root 2024-03-16T11:09:29.545 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-mds[66485]: -- 172.21.15.153:0/1955225754 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56314f27ba40 con 0x56314f24ec00 2024-03-16T11:09:29.545 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f[66481]: starting mds.f at 2024-03-16T11:09:29.545 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i[67102]: ignoring --setuser ceph since I am not root 2024-03-16T11:09:29.545 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i[67102]: ignoring --setgroup ceph since I am not root 2024-03-16T11:09:29.545 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-mds[67106]: -- 172.21.15.153:0/2546108750 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e6400b7a40 con 0x55e64008ac00 2024-03-16T11:09:29.545 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:09:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i[67102]: starting mds.i at 2024-03-16T11:09:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:29.700+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f261002d1b0 con 0x7f26101022b0 2024-03-16T11:09:29.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a[32231]: 2024-03-16T11:09:29.357+0000 7f1113964700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T11:09:30.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.362+0000 7f260cff9640 1 -- 172.21.15.78:0/3242961532 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) ==== 104+0+0 (secure 0 0 0) 0x7f260809bab0 con 0x7f26101022b0 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f25fc0a2800 msgr2=0x7f25fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f25fc0a2800 0x7f25fc0a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2600005d00 tx=0x7f2600005ef0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101022b0 msgr2=0x7f261019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101022b0 0x7f261019eff0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f260800eb10 tx=0x7f260800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 shutdown_connections 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26101022b0 0x7f261019eff0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26101018f0 0x7f2610197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f25fc0a2800 0x7f25fc0a4cc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 --2- 172.21.15.78:0/3242961532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2610100700 0x7f2610197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 >> 172.21.15.78:0/3242961532 conn(0x7f26100fbc90 msgr2=0x7f2610104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 shutdown_connections 2024-03-16T11:09:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.365+0000 7f2615590640 1 -- 172.21.15.78:0/3242961532 wait complete. 2024-03-16T11:09:30.380 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-16T11:09:30.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:30 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1168077676' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T11:09:30.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:30 smithi098 ceph-mon[35848]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T11:09:30.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:30 smithi098 ceph-mon[35848]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T11:09:30.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:30 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1168077676' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T11:09:30.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:30 smithi098 ceph-mon[35848]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T11:09:30.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:30 smithi098 ceph-mon[35848]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T11:09:30.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:30 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3242961532' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:09:30.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 -- 172.21.15.78:0/1557238710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc100700 msgr2=0x7fdbcc100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 --2- 172.21.15.78:0/1557238710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc100700 0x7fdbcc100b20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009900 tx=0x7fdbc002bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 -- 172.21.15.78:0/1557238710 shutdown_connections 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 --2- 172.21.15.78:0/1557238710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbcc1022b0 0x7fdbcc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 --2- 172.21.15.78:0/1557238710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc1018f0 0x7fdbcc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 --2- 172.21.15.78:0/1557238710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc100700 0x7fdbcc100b20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 -- 172.21.15.78:0/1557238710 >> 172.21.15.78:0/1557238710 conn(0x7fdbcc0fbc90 msgr2=0x7fdbcc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 -- 172.21.15.78:0/1557238710 shutdown_connections 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.486+0000 7fdbd3159640 1 -- 172.21.15.78:0/1557238710 wait complete. 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 Processor -- start 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 -- start start 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbcc100700 0x7fdbcc1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc1018f0 0x7fdbcc196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc1022b0 0x7fdbcc19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc10c280 con 0x7fdbcc1022b0 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc10c100 con 0x7fdbcc1018f0 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd3159640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc10c400 con 0x7fdbcc100700 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc1022b0 0x7fdbcc19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc1022b0 0x7fdbcc19df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33416/0 (socket says 172.21.15.78:33416) 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 -- 172.21.15.78:0/690172707 learned_addr learned my addr 172.21.15.78:0/690172707 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbcbfff640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc1018f0 0x7fdbcc196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd0ef5640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbcc100700 0x7fdbcc1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:30.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 -- 172.21.15.78:0/690172707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbcc100700 msgr2=0x7fdbcc1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbcc100700 0x7fdbcc1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 -- 172.21.15.78:0/690172707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc1018f0 msgr2=0x7fdbcc196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc1018f0 0x7fdbcc196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 -- 172.21.15.78:0/690172707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbcc19e5c0 con 0x7fdbcc1022b0 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbd16f6640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc1022b0 0x7fdbcc19df50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400b970 tx=0x7fdbc400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.487+0000 7fdbc9ffb640 1 -- 172.21.15.78:0/690172707 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4019070 con 0x7fdbcc1022b0 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.488+0000 7fdbc9ffb640 1 -- 172.21.15.78:0/690172707 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc4014440 con 0x7fdbcc1022b0 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.488+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbcc19e8b0 con 0x7fdbcc1022b0 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.488+0000 7fdbc9ffb640 1 -- 172.21.15.78:0/690172707 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40075d0 con 0x7fdbcc1022b0 2024-03-16T11:09:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.488+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbcc19ee80 con 0x7fdbcc1022b0 2024-03-16T11:09:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.489+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbcc107ac0 con 0x7fdbcc1022b0 2024-03-16T11:09:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.490+0000 7fdbc9ffb640 1 -- 172.21.15.78:0/690172707 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fdbc401c070 con 0x7fdbcc1022b0 2024-03-16T11:09:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.490+0000 7fdbc9ffb640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.490+0000 7fdbd0ef5640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.490+0000 7fdbc9ffb640 1 -- 172.21.15.78:0/690172707 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdbc40ce0b0 con 0x7fdbcc1022b0 2024-03-16T11:09:30.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.491+0000 7fdbd0ef5640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006010 tx=0x7fdbc0005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:30.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.495+0000 7fdbc9ffb640 1 -- 172.21.15.78:0/690172707 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdbc409a9f0 con 0x7fdbcc1022b0 2024-03-16T11:09:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:30 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1168077676' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T11:09:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:30 smithi153 ceph-mon[35521]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T11:09:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:30 smithi153 ceph-mon[35521]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T11:09:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:30 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1168077676' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T11:09:30.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:30 smithi153 ceph-mon[35521]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T11:09:30.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:30 smithi153 ceph-mon[35521]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T11:09:30.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:30 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3242961532' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:09:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:30 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1168077676' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T11:09:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:30 smithi078 ceph-mon[32235]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T11:09:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:30 smithi078 ceph-mon[32235]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T11:09:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:30 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1168077676' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T11:09:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:30 smithi078 ceph-mon[32235]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T11:09:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:30 smithi078 ceph-mon[32235]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T11:09:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:30 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3242961532' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:09:30.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:30.670+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdbcc0623e0 con 0x7fdbcc1022b0 2024-03-16T11:09:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.364+0000 7fdbc9ffb640 1 -- 172.21.15.78:0/690172707 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) ==== 181+0+0 (secure 0 0 0) 0x7fdbc409ea90 con 0x7fdbcc1022b0 2024-03-16T11:09:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-03-16T11:09:31.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdbbc0a2800 msgr2=0x7fdbbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006010 tx=0x7fdbc0005be0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc1022b0 msgr2=0x7fdbcc19df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc1022b0 0x7fdbcc19df50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400b970 tx=0x7fdbc400be30 comp rx=0 tx=0).stop 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 shutdown_connections 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbcc1022b0 0x7fdbcc19df50 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc1018f0 0x7fdbcc196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 --2- 172.21.15.78:0/690172707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbcc100700 0x7fdbcc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 >> 172.21.15.78:0/690172707 conn(0x7fdbcc0fbc90 msgr2=0x7fdbcc104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 shutdown_connections 2024-03-16T11:09:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.368+0000 7fdbd3159640 1 -- 172.21.15.78:0/690172707 wait complete. 2024-03-16T11:09:31.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-16T11:09:31.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: pgmap v455: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: Cluster is now healthy 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/3242961532' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6837/1298234916,v1:172.21.15.98:6839/1298234916] up:boot 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6834/2746850950,v1:172.21.15.153:6835/2746850950] up:boot 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6834/747621303,v1:172.21.15.98:6835/747621303] up:boot 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.98:6836/2192500100,v1:172.21.15.98:6838/2192500100] up:boot 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6836/2310015450,v1:172.21.15.153:6837/2310015450] up:boot 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: mds.? [v2:172.21.15.153:6838/467588638,v1:172.21.15.153:6839/467588638] up:boot 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: fsmap 12 up:standby 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:09:31.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:09:31.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:09:31.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:09:31.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:09:31.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:09:31.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:31 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/690172707' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T11:09:31.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1238061558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd8100700 msgr2=0x7f1fd8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:31.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1238061558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd8100700 0x7f1fd8100b20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009900 tx=0x7f1fc802bc30 comp rx=0 tx=0).stop 2024-03-16T11:09:31.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1238061558 shutdown_connections 2024-03-16T11:09:31.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1238061558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd81022b0 0x7f1fd8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1238061558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd81018f0 0x7f1fd8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1238061558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd8100700 0x7f1fd8100b20 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1238061558 >> 172.21.15.78:0/1238061558 conn(0x7f1fd80fbc90 msgr2=0x7f1fd80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1238061558 shutdown_connections 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1238061558 wait complete. 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.494+0000 7f1fdd93b640 1 Processor -- start 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 -- start start 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8100700 0x7f1fd8199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd81018f0 0x7f1fd819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd81022b0 0x7f1fd81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd810d350 con 0x7f1fd81018f0 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd810d1d0 con 0x7f1fd8100700 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd810d4d0 con 0x7f1fd81022b0 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd81018f0 0x7f1fd819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd81018f0 0x7f1fd819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33432/0 (socket says 172.21.15.78:33432) 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 -- 172.21.15.78:0/1828368249 learned_addr learned my addr 172.21.15.78:0/1828368249 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd77fe640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd81022b0 0x7f1fd81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd6ffd640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8100700 0x7f1fd8199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 -- 172.21.15.78:0/1828368249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd81022b0 msgr2=0x7f1fd81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd81022b0 0x7f1fd81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 -- 172.21.15.78:0/1828368249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8100700 msgr2=0x7f1fd8199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8100700 0x7f1fd8199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 -- 172.21.15.78:0/1828368249 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd81a1800 con 0x7f1fd81018f0 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd6ffd640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8100700 0x7f1fd8199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fd67fc640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd81018f0 0x7f1fd819a120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00c920 tx=0x7f1fcc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdc939640 1 -- 172.21.15.78:0/1828368249 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc007c70 con 0x7f1fd81018f0 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdc939640 1 -- 172.21.15.78:0/1828368249 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fcc007e10 con 0x7f1fd81018f0 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd81a1af0 con 0x7f1fd81018f0 2024-03-16T11:09:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.495+0000 7f1fdc939640 1 -- 172.21.15.78:0/1828368249 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc018820 con 0x7f1fd81018f0 2024-03-16T11:09:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.496+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd81a2090 con 0x7f1fd81018f0 2024-03-16T11:09:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.496+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd81a22c0 con 0x7f1fd81018f0 2024-03-16T11:09:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.497+0000 7f1fdc939640 1 -- 172.21.15.78:0/1828368249 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1fcc0189c0 con 0x7f1fd81018f0 2024-03-16T11:09:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.498+0000 7f1fdc939640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1fc00a2880 0x7f1fc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:09:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.498+0000 7f1fd6ffd640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1fc00a2880 0x7f1fc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:09:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.498+0000 7f1fdc939640 1 -- 172.21.15.78:0/1828368249 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8002+0+0 (secure 0 0 0) 0x7f1fcc0cea90 con 0x7f1fd81018f0 2024-03-16T11:09:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.498+0000 7f1fd6ffd640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1fc00a2880 0x7f1fc00a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80096c0 tx=0x7f1fc8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:09:31.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.502+0000 7f1fdc939640 1 -- 172.21.15.78:0/1828368249 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1fcc014030 con 0x7f1fd81018f0 2024-03-16T11:09:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: pgmap v455: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: Cluster is now healthy 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/3242961532' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6837/1298234916,v1:172.21.15.98:6839/1298234916] up:boot 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6834/2746850950,v1:172.21.15.153:6835/2746850950] up:boot 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6834/747621303,v1:172.21.15.98:6835/747621303] up:boot 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.98:6836/2192500100,v1:172.21.15.98:6838/2192500100] up:boot 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6836/2310015450,v1:172.21.15.153:6837/2310015450] up:boot 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: mds.? [v2:172.21.15.153:6838/467588638,v1:172.21.15.153:6839/467588638] up:boot 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: fsmap 12 up:standby 2024-03-16T11:09:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:09:31.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:09:31.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:09:31.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:09:31.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:09:31.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:09:31.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:31 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/690172707' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: pgmap v455: 97 pgs: 97 active+clean; 1.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: Cluster is now healthy 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/3242961532' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6837/1298234916,v1:172.21.15.98:6839/1298234916] up:boot 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6834/2746850950,v1:172.21.15.153:6835/2746850950] up:boot 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6834/747621303,v1:172.21.15.98:6835/747621303] up:boot 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.98:6836/2192500100,v1:172.21.15.98:6838/2192500100] up:boot 2024-03-16T11:09:31.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6836/2310015450,v1:172.21.15.153:6837/2310015450] up:boot 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: mds.? [v2:172.21.15.153:6838/467588638,v1:172.21.15.153:6839/467588638] up:boot 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: fsmap 12 up:standby 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='mgr.14152 172.21.15.78:0/1991572957' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:09:31.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:31 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/690172707' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T11:09:31.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:31.683+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1fd81a2490 con 0x7f1fd81018f0 2024-03-16T11:09:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.372+0000 7f1fdc939640 1 -- 172.21.15.78:0/1828368249 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) ==== 169+0+0 (secure 0 0 0) 0x7f1fcc00de20 con 0x7f1fd81018f0 2024-03-16T11:09:32.373 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-03-16T11:09:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1fc00a2880 msgr2=0x7f1fc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1fc00a2880 0x7f1fc00a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80096c0 tx=0x7f1fc8000bc0 comp rx=0 tx=0).stop 2024-03-16T11:09:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd81018f0 msgr2=0x7f1fd819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:09:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd81018f0 0x7f1fd819a120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00c920 tx=0x7f1fcc00cde0 comp rx=0 tx=0).stop 2024-03-16T11:09:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 shutdown_connections 2024-03-16T11:09:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:6800/4254906951,v1:172.21.15.78:6801/4254906951] conn(0x7f1fc00a2880 0x7f1fc00a4d40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd81022b0 0x7f1fd81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:32.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd81018f0 0x7f1fd819a120 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:32.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 --2- 172.21.15.78:0/1828368249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8100700 0x7f1fd8199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:09:32.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.374+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 >> 172.21.15.78:0/1828368249 conn(0x7f1fd80fbc90 msgr2=0x7f1fd8109500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:09:32.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.375+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 shutdown_connections 2024-03-16T11:09:32.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-16T11:09:32.375+0000 7f1fdd93b640 1 -- 172.21.15.78:0/1828368249 wait complete. 2024-03-16T11:09:32.393 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:09:32.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:09:32.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:09:32.485 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-16T11:09:32.485 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T11:09:32.500 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T11:09:32.516 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T11:09:32.542 INFO:tasks.cephadm:Stopping all daemons... 2024-03-16T11:09:32.542 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-16T11:09:32.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a 2024-03-16T11:09:32.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:32 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/690172707' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T11:09:32.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:32 smithi098 ceph-mon[35848]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T11:09:32.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:32 smithi098 ceph-mon[35848]: from='client.? 172.21.15.78:0/1828368249' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T11:09:32.715 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:32 smithi098 ceph-mon[35848]: pgmap v457: 65 pgs: 65 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:09:32.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:32 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/690172707' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T11:09:32.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:32 smithi153 ceph-mon[35521]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T11:09:32.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:32 smithi153 ceph-mon[35521]: from='client.? 172.21.15.78:0/1828368249' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T11:09:32.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:32 smithi153 ceph-mon[35521]: pgmap v457: 65 pgs: 65 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:09:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/690172707' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T11:09:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 ceph-mon[32235]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T11:09:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 ceph-mon[32235]: from='client.? 172.21.15.78:0/1828368249' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T11:09:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 ceph-mon[32235]: pgmap v457: 65 pgs: 65 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:09:32.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 systemd[1]: Stopping Ceph mon.a for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:33.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a[32231]: 2024-03-16T11:09:32.829+0000 7f111916f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:09:33.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a[32231]: 2024-03-16T11:09:32.829+0000 7f111916f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-16T11:09:33.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:32 smithi078 podman[91332]: 2024-03-16 11:09:32.973852919 +0000 UTC m=+0.237635622 container died b5c01b21f69c01bece854b37128e32a817931f264d2a627a234ae336acb0e572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-16T11:09:33.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:33 smithi078 podman[91332]: 2024-03-16 11:09:33.505558636 +0000 UTC m=+0.769341350 container cleanup b5c01b21f69c01bece854b37128e32a817931f264d2a627a234ae336acb0e572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:09:33.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:33 smithi078 bash[91332]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a 2024-03-16T11:09:33.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 16 11:09:33 smithi078 podman[91346]: 2024-03-16 11:09:33.706917987 +0000 UTC m=+0.737083922 container remove b5c01b21f69c01bece854b37128e32a817931f264d2a627a234ae336acb0e572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-03-16T11:09:33.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.a.service' 2024-03-16T11:09:33.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:33.973 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-16T11:09:33.973 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-16T11:09:33.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.b 2024-03-16T11:09:34.307 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:34 smithi098 systemd[1]: Stopping Ceph mon.b for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:34.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-b[35843]: 2024-03-16T11:09:34.306+0000 7feb4542d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:09:34.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-b[35843]: 2024-03-16T11:09:34.306+0000 7feb4542d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-16T11:09:34.714 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:34 smithi098 podman[70017]: 2024-03-16 11:09:34.511628601 +0000 UTC m=+0.335739770 container died bfbc5ea10822f23b3c1672f3eeee9f0741e6147062a16e0702f21f479add672f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-16T11:09:35.142 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:34 smithi098 podman[70017]: 2024-03-16 11:09:34.857249463 +0000 UTC m=+0.681360641 container cleanup bfbc5ea10822f23b3c1672f3eeee9f0741e6147062a16e0702f21f479add672f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-b, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:09:35.142 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 16 11:09:34 smithi098 bash[70017]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-b 2024-03-16T11:09:35.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.b.service' 2024-03-16T11:09:35.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:35.324 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-16T11:09:35.325 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-16T11:09:35.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.c 2024-03-16T11:09:35.679 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:35 smithi153 systemd[1]: Stopping Ceph mon.c for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:36.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c[35503]: 2024-03-16T11:09:35.679+0000 7f4924313700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:09:36.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c[35503]: 2024-03-16T11:09:35.679+0000 7f4924313700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-16T11:09:36.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:35 smithi153 podman[70187]: 2024-03-16 11:09:35.740791451 +0000 UTC m=+0.215330504 container died c9ce64dc1ae863f5c1aada3bb2704aa5bb45956b25abee34df8f4d7b2f95df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T11:09:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:36 smithi153 podman[70187]: 2024-03-16 11:09:36.129417488 +0000 UTC m=+0.603956539 container cleanup c9ce64dc1ae863f5c1aada3bb2704aa5bb45956b25abee34df8f4d7b2f95df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-16T11:09:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Mar 16 11:09:36 smithi153 bash[70187]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mon-c 2024-03-16T11:09:36.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mon.c.service' 2024-03-16T11:09:36.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:36.544 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-16T11:09:36.545 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-16T11:09:36.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.x 2024-03-16T11:09:36.934 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:09:36 smithi078 systemd[1]: Stopping Ceph mgr.x for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:37.194 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:09:36 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:09:36.934+0000 7f2ded1aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:09:37.194 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:09:36 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x[32470]: 2024-03-16T11:09:36.934+0000 7f2ded1aa700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-16T11:09:37.194 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:09:36 smithi078 podman[91453]: 2024-03-16 11:09:36.965759254 +0000 UTC m=+0.226265905 container died 564b1a81a389c86705be86b6639925309445ef1a31ac2dfe79d72f7eb2474002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-16T11:09:37.559 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:09:37 smithi078 podman[91453]: 2024-03-16 11:09:37.361270845 +0000 UTC m=+0.621777561 container cleanup 564b1a81a389c86705be86b6639925309445ef1a31ac2dfe79d72f7eb2474002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-16T11:09:37.559 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 16 11:09:37 smithi078 bash[91453]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-x 2024-03-16T11:09:37.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-16T11:09:37.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:37.808 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-16T11:09:37.808 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-16T11:09:37.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.y 2024-03-16T11:09:38.132 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 11:09:37 smithi098 systemd[1]: Stopping Ceph mgr.y for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:38.461 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 11:09:38 smithi098 podman[70141]: 2024-03-16 11:09:38.132813484 +0000 UTC m=+0.127829337 container died eef3db8b636f6c01f04a9d856bd79bbfbf7003abdb93323b5a22a74e29183d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:09:38.714 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 11:09:38 smithi098 podman[70141]: 2024-03-16 11:09:38.462172444 +0000 UTC m=+0.457188339 container cleanup eef3db8b636f6c01f04a9d856bd79bbfbf7003abdb93323b5a22a74e29183d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:09:38.715 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 11:09:38 smithi098 bash[70141]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y 2024-03-16T11:09:38.715 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 11:09:38 smithi098 podman[70155]: 2024-03-16 11:09:38.688672111 +0000 UTC m=+0.552269852 container remove eef3db8b636f6c01f04a9d856bd79bbfbf7003abdb93323b5a22a74e29183d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:09:38.715 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 16 11:09:38 smithi098 systemd[1]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-16T11:09:38.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-16T11:09:38.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:38.967 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-16T11:09:38.967 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-16T11:09:38.967 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.z 2024-03-16T11:09:39.293 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 11:09:39 smithi153 systemd[1]: Stopping Ceph mgr.z for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:39.601 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 11:09:39 smithi153 podman[70314]: 2024-03-16 11:09:39.31341821 +0000 UTC m=+0.151638009 container died 50074f2f014f5f19a067fa0d348f1f1ee9dfe5b318183c83c414749412e9555e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-16T11:09:39.927 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 11:09:39 smithi153 podman[70314]: 2024-03-16 11:09:39.684352403 +0000 UTC m=+0.522572209 container cleanup 50074f2f014f5f19a067fa0d348f1f1ee9dfe5b318183c83c414749412e9555e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T11:09:39.928 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Mar 16 11:09:39 smithi153 bash[70314]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mgr-z 2024-03-16T11:09:40.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-16T11:09:40.095 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:40.095 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-16T11:09:40.095 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-16T11:09:40.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.0 2024-03-16T11:09:40.524 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:40 smithi078 systemd[1]: Stopping Ceph osd.0 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:40.809 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:40 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0[46967]: 2024-03-16T11:09:40.523+0000 7f2fcf51d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:09:40.809 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:40 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0[46967]: 2024-03-16T11:09:40.523+0000 7f2fcf51d700 -1 osd.0 93 *** Got signal Terminated *** 2024-03-16T11:09:40.809 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:40 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0[46967]: 2024-03-16T11:09:40.523+0000 7f2fcf51d700 -1 osd.0 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:09:46.391 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:46 smithi078 podman[91577]: 2024-03-16 11:09:46.089711758 +0000 UTC m=+5.803391074 container died cfe5418bd94cedacb8f7995e7cffc45d58a27752f33e41aafc0f9fdcdb943cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-16T11:09:46.800 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:46 smithi078 podman[91577]: 2024-03-16 11:09:46.54116162 +0000 UTC m=+6.254840934 container cleanup cfe5418bd94cedacb8f7995e7cffc45d58a27752f33e41aafc0f9fdcdb943cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:09:46.800 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:46 smithi078 bash[91577]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0 2024-03-16T11:09:47.059 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:46 smithi078 podman[91594]: 2024-03-16 11:09:46.801060763 +0000 UTC m=+0.713806071 container remove cfe5418bd94cedacb8f7995e7cffc45d58a27752f33e41aafc0f9fdcdb943cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:09:47.335 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:47 smithi078 podman[91684]: 2024-03-16 11:09:47.05747317 +0000 UTC m=+0.017769106 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:09:47.335 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:47 smithi078 podman[91684]: 2024-03-16 11:09:47.202093773 +0000 UTC m=+0.162389700 container create 6c166501e3f7ec12aed34d2ecd9d7f950c1be9a237b86756a89587bc4d0cedd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T11:09:47.754 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:47 smithi078 podman[91684]: 2024-03-16 11:09:47.463581155 +0000 UTC m=+0.423877092 container init 6c166501e3f7ec12aed34d2ecd9d7f950c1be9a237b86756a89587bc4d0cedd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-16T11:09:47.755 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:47 smithi078 podman[91684]: 2024-03-16 11:09:47.467161097 +0000 UTC m=+0.427457026 container start 6c166501e3f7ec12aed34d2ecd9d7f950c1be9a237b86756a89587bc4d0cedd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T11:09:47.755 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:47 smithi078 podman[91684]: 2024-03-16 11:09:47.563311769 +0000 UTC m=+0.523607696 container attach 6c166501e3f7ec12aed34d2ecd9d7f950c1be9a237b86756a89587bc4d0cedd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-16T11:09:48.059 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 16 11:09:47 smithi078 podman[91684]: 2024-03-16 11:09:47.755193773 +0000 UTC m=+0.715489693 container died 6c166501e3f7ec12aed34d2ecd9d7f950c1be9a237b86756a89587bc4d0cedd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True) 2024-03-16T11:09:48.374 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.0.service' 2024-03-16T11:09:48.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:48.415 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-16T11:09:48.415 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-16T11:09:48.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.1 2024-03-16T11:09:48.714 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:48 smithi098 systemd[1]: Stopping Ceph osd.1 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:49.214 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1[40414]: 2024-03-16T11:09:48.733+0000 7fe18dc8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:09:49.214 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1[40414]: 2024-03-16T11:09:48.733+0000 7fe18dc8c700 -1 osd.1 93 *** Got signal Terminated *** 2024-03-16T11:09:49.215 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1[40414]: 2024-03-16T11:09:48.733+0000 7fe18dc8c700 -1 osd.1 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:09:54.386 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:54 smithi098 podman[70277]: 2024-03-16 11:09:54.114212385 +0000 UTC m=+5.506735595 container died fe6388661d71c9c55dc30ab2b480d841135755eddbe084da126baf99277994ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:09:54.714 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:54 smithi098 podman[70277]: 2024-03-16 11:09:54.492347067 +0000 UTC m=+5.884870276 container cleanup fe6388661d71c9c55dc30ab2b480d841135755eddbe084da126baf99277994ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:09:54.715 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:54 smithi098 bash[70277]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1 2024-03-16T11:09:55.062 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:54 smithi098 podman[70294]: 2024-03-16 11:09:54.752142969 +0000 UTC m=+0.637296102 container remove fe6388661d71c9c55dc30ab2b480d841135755eddbe084da126baf99277994ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:09:55.380 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:55 smithi098 podman[70389]: 2024-03-16 11:09:54.965676869 +0000 UTC m=+0.018511151 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:09:55.380 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:55 smithi098 podman[70389]: 2024-03-16 11:09:55.077997909 +0000 UTC m=+0.130832223 container create d81e01cae5de894aba42c4e1b891acb9ff9231c05a6c8bd3e57986d20906988c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:09:55.380 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:55 smithi098 podman[70389]: 2024-03-16 11:09:55.305662518 +0000 UTC m=+0.358496803 container init d81e01cae5de894aba42c4e1b891acb9ff9231c05a6c8bd3e57986d20906988c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-16T11:09:55.380 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:55 smithi098 podman[70389]: 2024-03-16 11:09:55.308888265 +0000 UTC m=+0.361722541 container start d81e01cae5de894aba42c4e1b891acb9ff9231c05a6c8bd3e57986d20906988c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T11:09:55.380 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:55 smithi098 podman[70389]: 2024-03-16 11:09:55.380364224 +0000 UTC m=+0.433198516 container attach d81e01cae5de894aba42c4e1b891acb9ff9231c05a6c8bd3e57986d20906988c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T11:09:55.714 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 16 11:09:55 smithi098 podman[70389]: 2024-03-16 11:09:55.599519678 +0000 UTC m=+0.652353951 container died d81e01cae5de894aba42c4e1b891acb9ff9231c05a6c8bd3e57986d20906988c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T11:09:56.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.1.service' 2024-03-16T11:09:56.196 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:09:56.196 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-16T11:09:56.196 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-16T11:09:56.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.2 2024-03-16T11:09:56.497 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:09:56 smithi153 systemd[1]: Stopping Ceph osd.2 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:09:56.793 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:09:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2[40666]: 2024-03-16T11:09:56.496+0000 7f13f7750700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:09:56.793 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:09:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2[40666]: 2024-03-16T11:09:56.496+0000 7f13f7750700 -1 osd.2 93 *** Got signal Terminated *** 2024-03-16T11:09:56.794 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:09:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2[40666]: 2024-03-16T11:09:56.496+0000 7f13f7750700 -1 osd.2 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:02.214 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:01 smithi153 podman[70454]: 2024-03-16 11:10:01.947843316 +0000 UTC m=+5.553194861 container died 5253383d8cbc6896b57e0398b93143bc6bf1c954b8ed7e6d86b5d92d2e6380cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T11:10:02.543 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:02 smithi153 podman[70454]: 2024-03-16 11:10:02.304932263 +0000 UTC m=+5.910283805 container cleanup 5253383d8cbc6896b57e0398b93143bc6bf1c954b8ed7e6d86b5d92d2e6380cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-16T11:10:02.543 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:02 smithi153 bash[70454]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2 2024-03-16T11:10:02.920 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:02 smithi153 podman[70473]: 2024-03-16 11:10:02.576260141 +0000 UTC m=+0.626996193 container remove 5253383d8cbc6896b57e0398b93143bc6bf1c954b8ed7e6d86b5d92d2e6380cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:10:03.209 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:02 smithi153 podman[70565]: 2024-03-16 11:10:02.823444653 +0000 UTC m=+0.017597760 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:03.209 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:02 smithi153 podman[70565]: 2024-03-16 11:10:02.940850393 +0000 UTC m=+0.135003487 container create 926ce97963b1c1cafb80d7aa71f99a2720b55ad382ea22080a976f5c7dfb5f33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T11:10:03.210 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:03 smithi153 podman[70565]: 2024-03-16 11:10:03.143174039 +0000 UTC m=+0.337327133 container init 926ce97963b1c1cafb80d7aa71f99a2720b55ad382ea22080a976f5c7dfb5f33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T11:10:03.210 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:03 smithi153 podman[70565]: 2024-03-16 11:10:03.14625793 +0000 UTC m=+0.340411027 container start 926ce97963b1c1cafb80d7aa71f99a2720b55ad382ea22080a976f5c7dfb5f33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-16T11:10:03.210 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:03 smithi153 podman[70565]: 2024-03-16 11:10:03.209738258 +0000 UTC m=+0.403891358 container attach 926ce97963b1c1cafb80d7aa71f99a2720b55ad382ea22080a976f5c7dfb5f33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:10:03.543 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:03 smithi153 podman[70565]: 2024-03-16 11:10:03.432839575 +0000 UTC m=+0.626992665 container died 926ce97963b1c1cafb80d7aa71f99a2720b55ad382ea22080a976f5c7dfb5f33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-16T11:10:03.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 16 11:10:03 smithi153 podman[70591]: 2024-03-16 11:10:03.946716821 +0000 UTC m=+0.508843515 container remove 926ce97963b1c1cafb80d7aa71f99a2720b55ad382ea22080a976f5c7dfb5f33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-16T11:10:03.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.2.service' 2024-03-16T11:10:04.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:04.009 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-16T11:10:04.009 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-16T11:10:04.010 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.3 2024-03-16T11:10:04.309 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:04 smithi078 systemd[1]: Stopping Ceph osd.3 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:04.809 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3[53953]: 2024-03-16T11:10:04.311+0000 7f5d78b0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:04.809 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3[53953]: 2024-03-16T11:10:04.311+0000 7f5d78b0e700 -1 osd.3 93 *** Got signal Terminated *** 2024-03-16T11:10:04.809 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:04 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3[53953]: 2024-03-16T11:10:04.311+0000 7f5d78b0e700 -1 osd.3 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:06.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:06 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:06.414+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:07.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:06 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:06.812+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:07.409 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:07 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:07.119+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:07.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:07.276+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:07.688 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:07 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:07.408+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:08.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:07.669+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:08.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:07 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:07.770+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:08.544 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:08.251+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:08.558 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:08 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:08.361+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:08.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:08 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:08.154+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:09.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:08.691+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:09.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:08 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:08.819+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:09.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:09.255+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:09.559 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:09 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:09.378+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:09.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:09 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:09.148+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:10.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:09.677+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:10.059 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:09 smithi078 podman[91793]: 2024-03-16 11:10:09.787757025 +0000 UTC m=+5.582433039 container died 8bc55634216e8bd0fe5dfd705f19bd0fbf2fad9603f54403f1b99c4ac6dfb64d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-16T11:10:10.215 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:09 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:09.842+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:10.427 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:10 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:10.129+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:10.427 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:10 smithi078 podman[91793]: 2024-03-16 11:10:10.31219991 +0000 UTC m=+6.106875931 container cleanup 8bc55634216e8bd0fe5dfd705f19bd0fbf2fad9603f54403f1b99c4ac6dfb64d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-16T11:10:10.427 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:10 smithi078 bash[91793]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3 2024-03-16T11:10:10.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:10.289+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:10.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:10 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:10.426+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:10.809 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:10 smithi078 podman[91822]: 2024-03-16 11:10:10.513627173 +0000 UTC m=+0.719182455 container remove 8bc55634216e8bd0fe5dfd705f19bd0fbf2fad9603f54403f1b99c4ac6dfb64d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-16T11:10:11.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:10.717+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:11.184 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:11 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:11.085+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:11.184 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:10 smithi078 podman[91914]: 2024-03-16 11:10:10.906252035 +0000 UTC m=+0.100121371 container create e27765af8cdf95cbacabf3b40d3acf0b39b352d901c94a22d4686312c86f0a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-16T11:10:11.184 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:10 smithi078 podman[91914]: 2024-03-16 11:10:10.824850754 +0000 UTC m=+0.018720099 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:11.184 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:11 smithi078 podman[91914]: 2024-03-16 11:10:11.100537322 +0000 UTC m=+0.294406659 container init e27765af8cdf95cbacabf3b40d3acf0b39b352d901c94a22d4686312c86f0a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:10:11.184 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:11 smithi078 podman[91914]: 2024-03-16 11:10:11.103735231 +0000 UTC m=+0.297604577 container start e27765af8cdf95cbacabf3b40d3acf0b39b352d901c94a22d4686312c86f0a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-16T11:10:11.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:10 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:10.861+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:11.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:11.242+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:11.559 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:11 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:11.424+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:11.559 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:11 smithi078 podman[91914]: 2024-03-16 11:10:11.183693897 +0000 UTC m=+0.377563255 container attach e27765af8cdf95cbacabf3b40d3acf0b39b352d901c94a22d4686312c86f0a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307) 2024-03-16T11:10:11.559 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 16 11:10:11 smithi078 podman[91914]: 2024-03-16 11:10:11.392852009 +0000 UTC m=+0.586721346 container died e27765af8cdf95cbacabf3b40d3acf0b39b352d901c94a22d4686312c86f0a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-03-16T11:10:12.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:11.745+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:12.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.3.service' 2024-03-16T11:10:12.101 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:12.101 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-16T11:10:12.101 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-16T11:10:12.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.4 2024-03-16T11:10:12.130 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:11 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:11.843+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:12.309 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:12 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:12.123+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:12.464 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:12 smithi098 systemd[1]: Stopping Ceph osd.4 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:12.464 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:12 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4[47345]: 2024-03-16T11:10:12.369+0000 7faddd2f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:12.465 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:12 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4[47345]: 2024-03-16T11:10:12.369+0000 7faddd2f0700 -1 osd.4 93 *** Got signal Terminated *** 2024-03-16T11:10:12.465 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:12 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4[47345]: 2024-03-16T11:10:12.369+0000 7faddd2f0700 -1 osd.4 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:12.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:12 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:12.206+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:12.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:12 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:12.437+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:13.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:12 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:12.785+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:13.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:12 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:12.891+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:13.485 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:13 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:13.107+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:13.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:13 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:13.227+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:13.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:13 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:13.484+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:14.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:13 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:13.738+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:14.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:13 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:13.928+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:14.498 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:14 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:14.096+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:14.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:14.227+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:14.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:14.412+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:14.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:14 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:14.498+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:15.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:14 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:14.740+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:15.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:14 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:14.897+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:15.511 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:15 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:15.126+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:15.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:15 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:15.229+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:15.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:15 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:15.394+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:15.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:15 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:15.511+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:16.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:15 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:15.706+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:16.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:15 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:15.904+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:16.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:15 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:15.904+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:16.490 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:16.097+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:16.491 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:16.097+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:16.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:16 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:16.224+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:16.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:16 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:16.440+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:16.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:16.490+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:16.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:16 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:16.490+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:17.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:16 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:16.745+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:17.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:16.931+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:17.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:16 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:16.931+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:17.428 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:17 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:17.207+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:17.428 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:17 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:17.207+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:17.542 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:17.062+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:17.542 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:17.062+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:17.758 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:17 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:17.428+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:17.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:17.538+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:17.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:17.538+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:18.004 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:17 smithi098 podman[70495]: 2024-03-16 11:10:17.724302864 +0000 UTC m=+5.431167494 container died f6e9968dc0603fdb05e006f997ce82ed71fc60de134dd0bd6f17de710b81357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-16T11:10:18.005 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:17 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:17.766+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:18.005 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:17 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:17.942+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:18.005 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:17 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:17.942+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:18.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:17 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:17.758+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:18.318 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:18.242+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:18.318 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:18.242+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:18.366 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:18 smithi098 podman[70495]: 2024-03-16 11:10:18.094916655 +0000 UTC m=+5.801781295 container cleanup f6e9968dc0603fdb05e006f997ce82ed71fc60de134dd0bd6f17de710b81357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T11:10:18.366 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:18 smithi098 bash[70495]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4 2024-03-16T11:10:18.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:18 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:18.097+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:18.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:18 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:18.097+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:18.684 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:18 smithi098 podman[70515]: 2024-03-16 11:10:18.363224934 +0000 UTC m=+0.636724669 container remove f6e9968dc0603fdb05e006f997ce82ed71fc60de134dd0bd6f17de710b81357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-16T11:10:18.711 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:18.382+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:18.965 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:18 smithi098 podman[70613]: 2024-03-16 11:10:18.590156883 +0000 UTC m=+0.019916068 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:18.965 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:18 smithi098 podman[70613]: 2024-03-16 11:10:18.718220588 +0000 UTC m=+0.147979746 container create 2b0c07a578b9a28ea0cb28fc9300cc5e0479b4872e7c23c4242db427a461c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T11:10:18.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:18 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:18.744+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:19.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:18.710+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:19.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:18 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:18.710+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:19.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:18 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:18.582+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:19.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:18 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:18.582+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:19.271 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:18 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:18.966+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:19.271 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:18 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:18.966+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:19.271 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:18 smithi098 podman[70613]: 2024-03-16 11:10:18.966665145 +0000 UTC m=+0.396424302 container init 2b0c07a578b9a28ea0cb28fc9300cc5e0479b4872e7c23c4242db427a461c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T11:10:19.271 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:18 smithi098 podman[70613]: 2024-03-16 11:10:18.970197678 +0000 UTC m=+0.399956842 container start 2b0c07a578b9a28ea0cb28fc9300cc5e0479b4872e7c23c4242db427a461c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T11:10:19.271 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:19 smithi098 podman[70613]: 2024-03-16 11:10:19.075147445 +0000 UTC m=+0.504906609 container attach 2b0c07a578b9a28ea0cb28fc9300cc5e0479b4872e7c23c4242db427a461c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T11:10:19.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:19 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:19.412+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:19.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:19 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:19.257+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:19.544 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:19 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:19.257+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:19.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:19.077+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:19.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:19.077+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:19.577 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 16 11:10:19 smithi098 podman[70613]: 2024-03-16 11:10:19.271625207 +0000 UTC m=+0.701384363 container died 2b0c07a578b9a28ea0cb28fc9300cc5e0479b4872e7c23c4242db427a461c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-16T11:10:19.837 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:19 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:19.738+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:19.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.4.service' 2024-03-16T11:10:19.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:19.940 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-16T11:10:19.940 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-16T11:10:19.940 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.5 2024-03-16T11:10:20.043 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:19 smithi153 systemd[1]: Stopping Ceph osd.5 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:20.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:19 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:19.750+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:20.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:19 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:19.750+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:20.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:19.561+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:20.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:19 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:19.561+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:20.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:20 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:20.000+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:20.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:20 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:20.000+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:20.524 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:20 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:20.069+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:20.524 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:20 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:20.069+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:20.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:20.458+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:20.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:20.248+0000 7fe3e6248700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:20.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:20.248+0000 7fe3e6248700 -1 osd.5 93 *** Got signal Terminated *** 2024-03-16T11:10:20.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:20.248+0000 7fe3e6248700 -1 osd.5 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:20.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:20.273+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:20.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:20.273+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:20.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:20 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:20.524+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:20.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:20 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:20.524+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:20.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:20 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:20.700+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:21.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:20.738+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:21.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:20 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:20.738+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:21.309 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:21 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:21.057+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:21.309 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:21 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:21.057+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:21.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:21 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:21.007+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:21.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:21 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:21.007+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:21.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:21 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:21.408+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:21.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:21 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:21.266+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:21.544 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:21 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:21.266+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:21.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:21 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:21.526+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:21.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:21 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:21.526+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:21.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:21 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:21.703+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:22.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:21 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:21.751+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:22.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:21 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:21.751+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:22.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:22 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:22.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:22.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:22 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:22.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:22.510 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:22 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:22.093+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:22.510 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:22 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:22.093+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:22.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:22 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:22.275+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:22.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:22 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:22.275+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:22.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:22 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:22.384+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:22.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:22 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:22.510+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:22.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:22 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:22.510+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:23.026 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:22 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:22.728+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:23.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:22 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:22.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:23.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:22 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:22.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:23.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:23 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:23.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:23.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:23 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:23.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:23.529 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:23 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:23.133+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:23.529 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:23 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:23.133+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:23.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:23 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:23.385+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:23.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:23 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:23.281+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:23.544 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:23 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:23.281+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:23.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:23 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:23.528+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:23.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:23 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:23.528+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:24.037 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:23 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:23.763+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:24.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:23 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:23.766+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:24.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:23 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:23.766+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:24.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:23 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:23.766+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:24.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:24 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:24.036+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:24.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:24 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:24.036+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:24.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:24 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:24.151+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:24.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:24 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:24.151+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:24.788 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:24.359+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:24.788 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:24.318+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.254969+0000 front 2024-03-16T11:09:42.255055+0000 (oldest deadline 2024-03-16T11:10:06.355022+0000) 2024-03-16T11:10:24.789 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5[47471]: 2024-03-16T11:10:24.318+0000 7fe3de834700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.956782+0000 front 2024-03-16T11:09:50.956904+0000 (oldest deadline 2024-03-16T11:10:16.856783+0000) 2024-03-16T11:10:25.032 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:24 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:24.721+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:25.033 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:24 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:24.721+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:25.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:24.788+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:25.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:24.788+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:25.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:24 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:24.788+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:25.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:24 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:24.574+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:25.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:24 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:24.574+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:25.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:24 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:24.574+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:25.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:25 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:25.032+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:25.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:25 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:25.032+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:25.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:25 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:25.032+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:25.541 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:25 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:25.166+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:25.541 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:25 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:25.166+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:25.710 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:25 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:25.310+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:25.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:25 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:25.541+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:25.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:25 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:25.541+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:25.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:25 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:25.541+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:25.973 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:25 smithi153 podman[70693]: 2024-03-16 11:10:25.710575888 +0000 UTC m=+5.580358145 container died b630a4727adb6f4c50b115fd6c83e2c1a9c182c1e3b897fc57bcb21bc669bea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T11:10:25.974 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:25 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:25.777+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:25.974 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:25 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:25.777+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:25.974 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:25 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:25.777+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:26.063 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:25 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:25.741+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:26.063 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:25 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:25.741+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:26.242 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:25 smithi153 podman[70693]: 2024-03-16 11:10:25.974145995 +0000 UTC m=+5.843928261 container cleanup b630a4727adb6f4c50b115fd6c83e2c1a9c182c1e3b897fc57bcb21bc669bea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:10:26.242 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:25 smithi153 bash[70693]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5 2024-03-16T11:10:26.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:26 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:26.062+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:26.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:26 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:26.062+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:26.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:26 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:26.062+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:26.543 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:26 smithi153 podman[70711]: 2024-03-16 11:10:26.242063564 +0000 UTC m=+0.527740028 container remove b630a4727adb6f4c50b115fd6c83e2c1a9c182c1e3b897fc57bcb21bc669bea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T11:10:26.543 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:26 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:26.355+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:26.558 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:26.149+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:26.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:26.149+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:26.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:26.149+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:26.903 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:26 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:26.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:26.903 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:26 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:26.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:26.903 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:26 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:26.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:26.903 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:26 smithi153 podman[70801]: 2024-03-16 11:10:26.492035072 +0000 UTC m=+0.018932927 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:26.903 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:26 smithi153 podman[70801]: 2024-03-16 11:10:26.593017725 +0000 UTC m=+0.119915612 container create 9e5dd5287e9adaad781f86ecaf8b9cd7b22e0aed13bca878c1efff4ce4390cae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T11:10:26.904 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:26 smithi153 podman[70801]: 2024-03-16 11:10:26.812046152 +0000 UTC m=+0.338943993 container init 9e5dd5287e9adaad781f86ecaf8b9cd7b22e0aed13bca878c1efff4ce4390cae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T11:10:26.904 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:26 smithi153 podman[70801]: 2024-03-16 11:10:26.815230998 +0000 UTC m=+0.342128841 container start 9e5dd5287e9adaad781f86ecaf8b9cd7b22e0aed13bca878c1efff4ce4390cae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:10:26.904 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:26 smithi153 podman[70801]: 2024-03-16 11:10:26.903590713 +0000 UTC m=+0.430488572 container attach 9e5dd5287e9adaad781f86ecaf8b9cd7b22e0aed13bca878c1efff4ce4390cae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:10:27.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:26.566+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:27.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:26.566+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:27.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:26.566+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:27.062 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:26 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:26.740+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:27.062 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:26 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:26.740+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:27.293 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 16 11:10:27 smithi153 podman[70801]: 2024-03-16 11:10:27.099854608 +0000 UTC m=+0.626752451 container died 9e5dd5287e9adaad781f86ecaf8b9cd7b22e0aed13bca878c1efff4ce4390cae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T11:10:27.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:27.062+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:27.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:27.062+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:27.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:27.062+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:27.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:27.184+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:27.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:27.184+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:27.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:27.184+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:27.657 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:27 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:27.345+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:27.657 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:27 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:27.345+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:27.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.5.service' 2024-03-16T11:10:27.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:27.721 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-16T11:10:27.721 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-16T11:10:27.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.6 2024-03-16T11:10:27.999 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:27.607+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:27.999 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:27.607+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:27.999 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:27.607+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:27.999 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:27 smithi078 systemd[1]: Stopping Ceph osd.6 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:28.000 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:27.998+0000 7f618f66b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:28.000 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:27.998+0000 7f618f66b700 -1 osd.6 93 *** Got signal Terminated *** 2024-03-16T11:10:28.000 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:27 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:27.998+0000 7f618f66b700 -1 osd.6 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:28.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:27 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:27.832+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:28.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:27 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:27.832+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:28.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:27 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:27.832+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:28.058 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:27.716+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:28.058 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:27 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:27.716+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:28.309 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:28.214+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:28.309 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:28.214+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:28.309 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:28.214+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:28.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:28 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:28.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:28.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:28 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:28.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:28.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:28 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:28.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:28.756 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:28 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:28.364+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:28.756 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:28 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:28.364+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:28.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:28 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:28.719+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:28.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:28 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:28.719+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:29.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:28 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:28.822+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:29.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:28 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:28.822+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:29.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:28 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:28.822+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:29.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:28.570+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:29.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:28.570+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:29.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:28 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:28.570+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:29.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:29.041+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:29.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:29.041+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:29.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:29.041+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:29.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:29.169+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:29.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:29.169+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:29.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:29.169+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:29.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:29.382+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:29.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:29.382+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:29.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:29.699+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:29.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:29 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:29.699+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:30.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:29.583+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:30.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:29.583+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:30.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:29 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:29.583+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:30.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:29.864+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:30.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:29.864+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:30.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:29 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:29.864+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:30.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:30 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:30.009+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:30.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:30 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:30.009+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:30.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:30 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:30.009+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:30.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:30.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:30.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:30.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:30.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:30.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:30.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:30 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:30.373+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:30.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:30 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:30.373+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:30.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:30 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:30.656+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:30.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:30 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:30.656+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:31.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:30.560+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:31.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:30.560+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:31.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:30 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:30.560+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:31.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:30 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:30.841+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:31.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:30 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:30.841+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:31.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:30 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:30.841+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:31.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:31.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:31.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:31.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:31.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:31.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:31.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:31.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:31.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:31.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:31.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:31.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:31.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:31.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:31.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:31.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:31.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:31 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:31.360+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:31.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:31 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:31.360+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:31.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:31.685+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:31.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:31.685+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:31.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:31.685+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:32.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:31.603+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:32.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:31.603+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:32.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:31.603+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:32.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:31 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:31.603+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.053762+0000 front 2024-03-16T11:10:09.053670+0000 (oldest deadline 2024-03-16T11:10:30.753795+0000) 2024-03-16T11:10:32.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:31 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:31.805+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:32.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:31 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:31.805+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:32.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:31 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:31.805+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:32.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:32 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:32.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:32.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:32 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:32.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:32.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:32 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:32.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:32.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:32 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:32.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:32.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:32.256+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:32.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:32.256+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:32.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:32.256+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:32.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:32.256+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:32.774 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:32 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:32.396+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:32.774 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:32 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:32.396+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:32.774 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:32 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:32.396+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:32.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:32 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:32.637+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:32.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:32 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:32.637+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:32.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:32 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:32.637+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:33.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:32 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:32.774+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:33.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:32 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:32.774+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:33.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:32 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:32.774+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:33.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:32.641+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:41.650011+0000 front 2024-03-16T11:09:41.650043+0000 (oldest deadline 2024-03-16T11:10:06.350053+0000) 2024-03-16T11:10:33.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:32.641+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.451055+0000 front 2024-03-16T11:09:50.450952+0000 (oldest deadline 2024-03-16T11:10:16.351037+0000) 2024-03-16T11:10:33.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:32.641+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.752796+0000 front 2024-03-16T11:09:59.752879+0000 (oldest deadline 2024-03-16T11:10:23.852630+0000) 2024-03-16T11:10:33.059 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:32 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6[60964]: 2024-03-16T11:10:32.641+0000 7f618bc5f700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.053762+0000 front 2024-03-16T11:10:09.053670+0000 (oldest deadline 2024-03-16T11:10:30.753795+0000) 2024-03-16T11:10:33.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:33 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:33.015+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:33.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:33 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:33.015+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:33.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:33 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:33.015+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:33.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:33 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:33.015+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:33.559 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:33 smithi078 podman[92019]: 2024-03-16 11:10:33.331818912 +0000 UTC m=+5.422163003 container died 8c62e68d57385333f1655b6def7ad37c9c30c1fd5df3cc11ffa2a56ae14e4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:10:33.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:33 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:33.222+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:33.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:33 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:33.222+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:33.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:33 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:33.222+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:33.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:33 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:33.222+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:33.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:33 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:33.380+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:33.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:33 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:33.380+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:33.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:33 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:33.380+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:33.958 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:33 smithi078 podman[92019]: 2024-03-16 11:10:33.774184541 +0000 UTC m=+5.864528683 container cleanup 8c62e68d57385333f1655b6def7ad37c9c30c1fd5df3cc11ffa2a56ae14e4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-16T11:10:33.958 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:33 smithi078 bash[92019]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6 2024-03-16T11:10:33.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:33 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:33.602+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:33.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:33 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:33.602+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:33.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:33 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:33.602+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:34.209 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:33 smithi078 podman[92036]: 2024-03-16 11:10:33.958682141 +0000 UTC m=+0.626538839 container remove 8c62e68d57385333f1655b6def7ad37c9c30c1fd5df3cc11ffa2a56ae14e4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T11:10:34.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:33 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:33.811+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:34.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:33 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:33.811+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:34.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:33 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:33.811+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:34.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:34.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:34.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:34.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:34.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:34.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:34.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:34.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:34.491 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:34 smithi078 podman[92129]: 2024-03-16 11:10:34.280408526 +0000 UTC m=+0.017000857 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:34.491 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:34 smithi078 podman[92129]: 2024-03-16 11:10:34.39301922 +0000 UTC m=+0.129611550 container create 71e33054ae4f3e65d64dc6833877a974de2bccc037d02aaa0193d988043830ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T11:10:34.491 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:34.208+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:34.491 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:34.208+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:34.492 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:34.208+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:34.492 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:34 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:34.208+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:34.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:34 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:34.398+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:34.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:34 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:34.398+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:34.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:34 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:34.398+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:34.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:34 smithi078 podman[92129]: 2024-03-16 11:10:34.612205468 +0000 UTC m=+0.348797804 container init 71e33054ae4f3e65d64dc6833877a974de2bccc037d02aaa0193d988043830ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-16T11:10:34.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:34 smithi078 podman[92129]: 2024-03-16 11:10:34.61525983 +0000 UTC m=+0.351852157 container start 71e33054ae4f3e65d64dc6833877a974de2bccc037d02aaa0193d988043830ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T11:10:34.809 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:34 smithi078 podman[92129]: 2024-03-16 11:10:34.695676914 +0000 UTC m=+0.432269259 container attach 71e33054ae4f3e65d64dc6833877a974de2bccc037d02aaa0193d988043830ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:10:34.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:34.609+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:34.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:34.609+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:34.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:34.609+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:35.198 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 16 11:10:34 smithi078 podman[92129]: 2024-03-16 11:10:34.901778404 +0000 UTC m=+0.638370719 container died 71e33054ae4f3e65d64dc6833877a974de2bccc037d02aaa0193d988043830ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T11:10:35.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:34 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:34.859+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:35.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:34 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:34.859+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:35.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:34 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:34.859+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:35.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:35.057+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:35.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:35.057+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:35.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:35.057+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:35.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:35.057+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:35.557 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:35 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:35.199+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:35.557 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:35 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:35.199+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:35.557 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:35 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:35.199+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:35.558 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:35 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:35.199+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:35.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.6.service' 2024-03-16T11:10:35.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:35.627 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-16T11:10:35.627 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-16T11:10:35.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.7 2024-03-16T11:10:35.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:35.362+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:35.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:35.362+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:35.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:35.362+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:35.925 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:35.641+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:35.926 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:35.641+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:35.926 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:35.641+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:35.926 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:35 smithi098 systemd[1]: Stopping Ceph osd.7 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:35.926 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:35.924+0000 7fa1190db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:36.214 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:35.924+0000 7fa1190db700 -1 osd.7 93 *** Got signal Terminated *** 2024-03-16T11:10:36.214 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:35 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:35.924+0000 7fa1190db700 -1 osd.7 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:36.215 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:36 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:36.012+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:36.215 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:36 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:36.012+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:36.215 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:36 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:36.012+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:36.215 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:36 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:36.012+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:36.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:35.891+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:36.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:35.891+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:36.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:35 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:35.891+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:36.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:36 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:36.211+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:36.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:36 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:36.211+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:36.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:36 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:36.211+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:36.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:36 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:36.211+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:36.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:36 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:36.378+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:36.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:36 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:36.378+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:36.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:36 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:36.378+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:36.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:36 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:36.652+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:36.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:36 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:36.652+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:36.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:36 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:36.652+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:37.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:36 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:36.924+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:37.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:36 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:36.924+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:37.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:36 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:36.924+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:37.461 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:37 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:37.186+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:37.461 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:37 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:37.186+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:37.461 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:37 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:37.186+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:37.461 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:37 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:37.186+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:37.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:37 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:37.032+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:37.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:37 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:37.032+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:37.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:37 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:37.032+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:37.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:37 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:37.032+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:37.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:37 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:37.342+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:37.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:37 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:37.342+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:37.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:37 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:37.342+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:37.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:37 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:37.691+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:37.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:37 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:37.691+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:37.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:37 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:37.691+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:38.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:37 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:37.883+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:38.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:37 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:37.883+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:38.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:37 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:37.883+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:38.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.044+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:38.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.044+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:38.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.044+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:38.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.044+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:38.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:38 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:38.141+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:38.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:38 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:38.141+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:38.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:38 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:38.141+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:38.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:38 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:38.141+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:38.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:38.392+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:38.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:38.392+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:38.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:38.392+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:38.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:38.392+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:38.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:38.671+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:38.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:38.671+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:38.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:38.671+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:39.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:38.839+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:39.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:38.839+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:39.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:38 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:38.839+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:39.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.998+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:39.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.998+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:39.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.998+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:39.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:38.998+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:39.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:39.157+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:39.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:39.157+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:39.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:39.157+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:39.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:39 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:39.157+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:39.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:39 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:39.389+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:39.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:39 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:39.389+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:39.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:39 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:39.389+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:39.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:39 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:39.389+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:39.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:39 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:39.626+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:39.969 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:39 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:39.626+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:39.969 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:39 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:39.626+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:40.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:39 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:39.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:40.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:39 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:39.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:40.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:39 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:39.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:40.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:40.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:40.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:40.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:40.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:40.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:40.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:40.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:40.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:40.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:40.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:40 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:40.202+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:40.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:40 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:40.202+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:40.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:40 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:40.202+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:40.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:40 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:40.202+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:40.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:40 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:40.425+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:40.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:40 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:40.425+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:40.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:40 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:40.425+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:40.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:40 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:40.425+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:40.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:40.626+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.378240+0000 front 2024-03-16T11:09:52.378220+0000 (oldest deadline 2024-03-16T11:10:17.078378+0000) 2024-03-16T11:10:40.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:40.626+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:09:59.880172+0000 front 2024-03-16T11:09:59.880048+0000 (oldest deadline 2024-03-16T11:10:24.580175+0000) 2024-03-16T11:10:40.965 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:40 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7[54335]: 2024-03-16T11:10:40.626+0000 7fa1156cf700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:09.281359+0000 front 2024-03-16T11:10:09.281326+0000 (oldest deadline 2024-03-16T11:10:30.981116+0000) 2024-03-16T11:10:41.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:40 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:40.842+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:41.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:40 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:40.842+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:41.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:40 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:40.842+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:41.350 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:41 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:41.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:41.350 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:41 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:41.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:41.350 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:41 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:41.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:41.350 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:41 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:41.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:41.350 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:41 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:41.025+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:41.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:41 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:41.162+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:41.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:41 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:41.162+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:41.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:41 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:41.162+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:41.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:41 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:41.162+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:41.675 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:41 smithi098 podman[70722]: 2024-03-16 11:10:41.350260674 +0000 UTC m=+5.527663632 container stop 35bc74da04eb2f18b6593a7b3d15e736a3e0f7db0aaaa79ed39b9544c50ae912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-16T11:10:41.675 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:41 smithi098 podman[70722]: 2024-03-16 11:10:41.376091454 +0000 UTC m=+5.553494412 container died 35bc74da04eb2f18b6593a7b3d15e736a3e0f7db0aaaa79ed39b9544c50ae912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-16T11:10:41.675 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:41 smithi098 podman[70722]: 2024-03-16 11:10:41.675408445 +0000 UTC m=+5.852811403 container cleanup 35bc74da04eb2f18b6593a7b3d15e736a3e0f7db0aaaa79ed39b9544c50ae912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-16T11:10:41.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:41.424+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:41.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:41.424+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:41.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:41.424+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:41.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:41.424+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:41.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:41 smithi098 bash[70722]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7 2024-03-16T11:10:41.964 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:41 smithi098 podman[70739]: 2024-03-16 11:10:41.885134011 +0000 UTC m=+0.528626204 container remove 35bc74da04eb2f18b6593a7b3d15e736a3e0f7db0aaaa79ed39b9544c50ae912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:10:42.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:41.821+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:42.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:41.821+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:42.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:41.821+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:42.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:41 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:41.821+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:42.399 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:42 smithi098 podman[70831]: 2024-03-16 11:10:42.191166281 +0000 UTC m=+0.018606542 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:42.399 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:42 smithi098 podman[70831]: 2024-03-16 11:10:42.294279223 +0000 UTC m=+0.121719480 container create e1cd5c9a3c2716fce2a8d92f8e4818ccd8ff51b48319fa9acddd6e29f9bd2352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-16T11:10:42.400 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:42 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:42.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:42.400 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:42 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:42.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:42.400 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:42 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:42.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:42.400 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:42 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:42.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:42.400 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:42 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:42.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:42.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:42 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:42.144+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:42.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:42 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:42.144+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:42.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:42 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:42.144+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:42.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:42 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:42.144+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:42.715 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:42 smithi098 podman[70831]: 2024-03-16 11:10:42.598628314 +0000 UTC m=+0.426068574 container init e1cd5c9a3c2716fce2a8d92f8e4818ccd8ff51b48319fa9acddd6e29f9bd2352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T11:10:42.716 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:42 smithi098 podman[70831]: 2024-03-16 11:10:42.604212401 +0000 UTC m=+0.431652658 container start e1cd5c9a3c2716fce2a8d92f8e4818ccd8ff51b48319fa9acddd6e29f9bd2352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:10:42.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:42.377+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:42.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:42.377+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:42.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:42.377+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:42.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:42.377+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:43.084 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:42 smithi098 podman[70831]: 2024-03-16 11:10:42.76405911 +0000 UTC m=+0.591499367 container attach e1cd5c9a3c2716fce2a8d92f8e4818ccd8ff51b48319fa9acddd6e29f9bd2352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T11:10:43.084 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 16 11:10:42 smithi098 podman[70831]: 2024-03-16 11:10:42.889693288 +0000 UTC m=+0.717133539 container died e1cd5c9a3c2716fce2a8d92f8e4818ccd8ff51b48319fa9acddd6e29f9bd2352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-16T11:10:43.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:42.850+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:43.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:42.850+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:43.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:42.850+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:43.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:42 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:42.850+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:43.359 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:43 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:43.083+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:43.359 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:43 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:43.083+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:43.359 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:43 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:43.083+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:43.359 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:43 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:43.083+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:43.360 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:43 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:43.083+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:43.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.7.service' 2024-03-16T11:10:43.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:43.426 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-16T11:10:43.426 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-16T11:10:43.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.8 2024-03-16T11:10:43.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:43 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:43.189+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:43.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:43 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:43.189+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:43.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:43 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:43.189+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:43.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:43 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:43.189+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:43.742 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:43.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:43.743 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:43.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:43.743 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:43.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:43.743 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:43.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:43.743 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 systemd[1]: Stopping Ceph osd.8 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:44.043 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:43.742+0000 7f55da6d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:44.043 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:43.742+0000 7f55da6d5700 -1 osd.8 93 *** Got signal Terminated *** 2024-03-16T11:10:44.043 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:43.742+0000 7f55da6d5700 -1 osd.8 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:44.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:43.883+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:44.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:43.883+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:44.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:43.883+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:44.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:43 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:43.883+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:44.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:44 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:44.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:44.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:44 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:44.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:44.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:44 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:44.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:44.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:44 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:44.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:44.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:44 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:44.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:44.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:44 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:44.169+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:44.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:44 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:44.169+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:44.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:44 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:44.169+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:44.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:44 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:44.169+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:44.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:44.333+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:44.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:44.333+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:44.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:44.333+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:44.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:44.333+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:45.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:44.844+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:45.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:44.844+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:45.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:44.844+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:45.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:44.844+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:45.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:45 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:45.047+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:45.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:45 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:45.047+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:45.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:45 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:45.047+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:45.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:45 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:45.047+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:45.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:45 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:45.047+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:45.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:45 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:45.201+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:45.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:45 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:45.201+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:45.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:45 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:45.201+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:45.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:45 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:45.201+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:45.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:45.352+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:45.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:45.352+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:45.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:45.352+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:45.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:45.352+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:46.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:45.819+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:46.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:45.819+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:46.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:45.819+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:46.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:45 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:45.819+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:46.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:46 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:46.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:46.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:46 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:46.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:46.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:46 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:46.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:46.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:46 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:46.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:46.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:46 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:46.058+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:46.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:46 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:46.204+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:46.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:46 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:46.204+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:46.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:46 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:46.204+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:46.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:46 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:46.204+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:46.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:46.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:46.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:46.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:46.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:46.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:46.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:46.337+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:47.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:46.814+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:47.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:46.814+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:47.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:46.814+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:47.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:46.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:47.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:46 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:46.815+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:47.366 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:47 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:47.027+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:47.366 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:47 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:47.027+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:47.366 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:47 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:47.027+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:47.366 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:47 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:47.027+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:47.366 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:47 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:47.027+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:47.366 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:47 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:47.027+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:47.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:47 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:47.244+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:47.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:47 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:47.244+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:47.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:47 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:47.244+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:47.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:47 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:47.244+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:47.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:47.324+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:47.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:47.324+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:47.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:47.324+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:47.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:47.324+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:48.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:47.852+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:48.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:47.852+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:48.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:47.852+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:48.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:47.852+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:48.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:47 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:47.852+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:48.376 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:48.073+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:48.376 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:48.073+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:48.376 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:48.073+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:48.376 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:48.073+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:48.376 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:48.073+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:48.376 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:48 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:48.073+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:48.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:48 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:48.215+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:48.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:48 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:48.215+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:48.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:48 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:48.215+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:48.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:48 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:48.215+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:48.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:48 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:48.215+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:48.568 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:48.305+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.737783+0000 front 2024-03-16T11:09:50.737826+0000 (oldest deadline 2024-03-16T11:10:14.237853+0000) 2024-03-16T11:10:48.568 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:48.305+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.239401+0000 front 2024-03-16T11:10:01.239655+0000 (oldest deadline 2024-03-16T11:10:26.539569+0000) 2024-03-16T11:10:48.568 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:48.305+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:06.540045+0000 front 2024-03-16T11:10:06.540194+0000 (oldest deadline 2024-03-16T11:10:31.840031+0000) 2024-03-16T11:10:48.568 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:48.305+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:15.341250+0000 front 2024-03-16T11:10:15.341112+0000 (oldest deadline 2024-03-16T11:10:38.240971+0000) 2024-03-16T11:10:48.568 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8[54304]: 2024-03-16T11:10:48.305+0000 7f55d6cc9700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.642629+0000 front 2024-03-16T11:10:24.642661+0000 (oldest deadline 2024-03-16T11:10:47.542728+0000) 2024-03-16T11:10:49.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:48.806+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:49.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:48.806+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:49.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:48.806+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:49.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:48.806+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:49.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:48 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:48.806+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:49.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:49 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:49.074+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:49.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:49 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:49.074+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:49.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:49 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:49.074+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:49.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:49 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:49.074+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:49.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:49 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:49.074+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:49.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:49 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:49.074+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:49.510 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:49 smithi153 podman[70906]: 2024-03-16 11:10:49.231623258 +0000 UTC m=+5.601219035 container died 9d05dc8d1b0eb96da8e57d5110baa17442ac2a604b6b8f5e3d4e1f0e64bd52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T11:10:49.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:49 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:49.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:49.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:49 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:49.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:49.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:49 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:49.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:49.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:49 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:49.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:49.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:49 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:49.218+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:49.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:49 smithi153 podman[70906]: 2024-03-16 11:10:49.600970084 +0000 UTC m=+5.970565863 container cleanup 9d05dc8d1b0eb96da8e57d5110baa17442ac2a604b6b8f5e3d4e1f0e64bd52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T11:10:49.794 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:49 smithi153 bash[70906]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8 2024-03-16T11:10:50.153 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:49 smithi153 podman[70934]: 2024-03-16 11:10:49.827290099 +0000 UTC m=+0.594627644 container remove 9d05dc8d1b0eb96da8e57d5110baa17442ac2a604b6b8f5e3d4e1f0e64bd52b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T11:10:50.153 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:49 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:49.849+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:50.153 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:49 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:49.849+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:50.153 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:49 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:49.849+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:50.154 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:49 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:49.849+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:50.154 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:49 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:49.849+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:50.405 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:50 smithi153 podman[71027]: 2024-03-16 11:10:50.153068641 +0000 UTC m=+0.111974410 container create a874adaa8f007f5dc06c6bd7cf972e60302c629f452ecfa501fd8d463b7bb1bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T11:10:50.405 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:50 smithi153 podman[71027]: 2024-03-16 11:10:50.0591076 +0000 UTC m=+0.018013340 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:50.405 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:50 smithi153 podman[71027]: 2024-03-16 11:10:50.322027987 +0000 UTC m=+0.280933714 container init a874adaa8f007f5dc06c6bd7cf972e60302c629f452ecfa501fd8d463b7bb1bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-16T11:10:50.406 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:50 smithi153 podman[71027]: 2024-03-16 11:10:50.325046142 +0000 UTC m=+0.283951870 container start a874adaa8f007f5dc06c6bd7cf972e60302c629f452ecfa501fd8d463b7bb1bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:10:50.406 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:50 smithi153 podman[71027]: 2024-03-16 11:10:50.405317678 +0000 UTC m=+0.364223436 container attach a874adaa8f007f5dc06c6bd7cf972e60302c629f452ecfa501fd8d463b7bb1bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T11:10:50.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:50 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:50.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:50.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:50 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:50.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:50.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:50 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:50.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:50.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:50 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:50.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:50.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:50 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:50.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:50.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:50 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:50.081+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:50.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:50 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:50.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:50.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:50 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:50.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:50.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:50 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:50.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:50.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:50 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:50.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:50.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:50 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:50.216+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:50.793 INFO:journalctl@ceph.osd.8.smithi153.stdout:Mar 16 11:10:50 smithi153 podman[71027]: 2024-03-16 11:10:50.612077718 +0000 UTC m=+0.570983455 container died a874adaa8f007f5dc06c6bd7cf972e60302c629f452ecfa501fd8d463b7bb1bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:10:51.233 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:50 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:50.865+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:51.234 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:50 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:50.865+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:51.234 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:50 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:50.865+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:51.234 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:50 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:50.865+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:51.234 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:50 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:50.865+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:51.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.8.service' 2024-03-16T11:10:51.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:51.300 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-16T11:10:51.300 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-16T11:10:51.300 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.9 2024-03-16T11:10:51.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:51 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:51.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:51.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:51 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:51.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:51.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:51 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:51.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:51.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:51 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:51.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:51.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:51 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:51.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:51.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:51 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:51.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:51.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.263+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:51.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.263+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:51.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.263+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:51.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.263+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:51.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.263+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:51.560 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 systemd[1]: Stopping Ceph osd.9 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:52.059 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.593+0000 7f15bf048700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:52.059 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.593+0000 7f15bf048700 -1 osd.9 93 *** Got signal Terminated *** 2024-03-16T11:10:52.059 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:51 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:51.593+0000 7f15bf048700 -1 osd.9 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:52.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:51 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:51.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:52.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:51 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:51.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:52.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:51 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:51.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:52.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:51 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:51.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:52.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:51 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:51.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:52.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:52 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:52.107+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:52.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:52 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:52.107+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:52.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:52 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:52.107+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:52.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:52 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:52.107+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:52.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:52 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:52.107+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:52.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:52 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:52.107+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:52.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:52 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:52.312+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:52.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:52 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:52.312+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:52.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:52 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:52.312+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:52.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:52 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:52.312+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:52.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:52 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:52.312+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:53.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:52 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:52.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:53.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:52 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:52.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:53.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:52 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:52.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:53.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:52 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:52.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:53.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:52 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:52.856+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:53.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:53 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:53.120+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:53.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:53 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:53.120+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:53.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:53 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:53.120+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:53.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:53 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:53.120+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:53.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:53 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:53.120+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:53.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:53 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:53.120+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:53.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:53 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:53.330+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:53.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:53 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:53.330+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:53.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:53 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:53.330+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:53.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:53 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:53.330+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:53.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:53 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:53.330+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:54.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:53 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:53.896+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:54.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:53 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:53.896+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:54.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:53 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:53.896+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:54.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:53 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:53.896+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:54.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:53 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:53.896+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:54.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:53 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:53.896+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:10:54.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:54.071+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:54.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:54.071+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:54.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:54.071+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:54.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:54.071+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:54.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:54.071+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:54.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:54.071+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:54.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:54 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:54.071+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:10:54.810 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:54.322+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:54.810 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:54.322+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:54.810 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:54.322+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:54.810 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:54.322+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:54.810 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:54 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:54.322+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:55.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:54.851+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:55.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:54.851+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:55.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:54.851+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:55.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:54.851+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:55.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:54.851+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:55.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:54.851+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:10:55.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:55 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:55.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:55.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:55 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:55.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:55.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:55 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:55.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:55.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:55 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:55.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:55.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:55 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:55.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:55.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:55 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:55.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:55.466 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:55 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:55.089+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:10:55.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:55.301+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:55.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:55.301+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:55.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:55.301+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:55.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:55.301+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:55.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:55 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:55.301+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:56.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:55 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:55.871+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:56.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:55 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:55.871+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:56.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:55 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:55.871+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:56.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:55 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:55.871+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:56.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:55 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:55.871+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:56.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:55 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:55.871+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:10:56.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:56 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:56.065+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:56.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:56 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:56.065+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:56.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:56 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:56.065+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:56.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:56 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:56.065+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:56.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:56 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:56.065+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:56.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:56 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:56.065+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:56.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:56 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:56.065+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:10:56.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:56.344+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:44.468486+0000 front 2024-03-16T11:09:44.468708+0000 (oldest deadline 2024-03-16T11:10:06.168739+0000) 2024-03-16T11:10:56.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:56.344+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:50.770243+0000 front 2024-03-16T11:09:50.770290+0000 (oldest deadline 2024-03-16T11:10:15.470332+0000) 2024-03-16T11:10:56.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:56.344+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:00.171457+0000 front 2024-03-16T11:10:00.171513+0000 (oldest deadline 2024-03-16T11:10:25.971830+0000) 2024-03-16T11:10:56.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:56.344+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:05.972243+0000 front 2024-03-16T11:10:05.972430+0000 (oldest deadline 2024-03-16T11:10:30.672322+0000) 2024-03-16T11:10:56.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:56 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9[67597]: 2024-03-16T11:10:56.344+0000 7f15b7634700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.075012+0000 front 2024-03-16T11:10:24.074865+0000 (oldest deadline 2024-03-16T11:10:47.574777+0000) 2024-03-16T11:10:57.185 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:56 smithi078 podman[92247]: 2024-03-16 11:10:56.933996605 +0000 UTC m=+5.440121721 container died 4759b92067e5784dda1ff90768a6141aa800a9d14bb51493591001f9d88d1bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T11:10:57.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:56.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:57.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:56.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:57.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:56.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:57.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:56.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:57.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:56.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:57.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:56 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:56.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:10:57.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:57 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:57.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:57.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:57 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:57.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:57.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:57 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:57.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:57.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:57 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:57.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:57.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:57 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:57.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:57.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:57 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:57.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:57.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:57 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:57.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:10:57.544 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:57 smithi078 podman[92247]: 2024-03-16 11:10:57.352007651 +0000 UTC m=+5.858132764 container cleanup 4759b92067e5784dda1ff90768a6141aa800a9d14bb51493591001f9d88d1bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-16T11:10:57.544 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:57 smithi078 bash[92247]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9 2024-03-16T11:10:57.809 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:57 smithi078 podman[92264]: 2024-03-16 11:10:57.544973818 +0000 UTC m=+0.606029468 container remove 4759b92067e5784dda1ff90768a6141aa800a9d14bb51493591001f9d88d1bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T11:10:58.139 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:57 smithi078 podman[92354]: 2024-03-16 11:10:57.837489558 +0000 UTC m=+0.108618373 container create d1295b611059406d3e8b14a4959f4dbb4a4facb28c2bd75393dd436c41875e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:10:58.139 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:57 smithi078 podman[92354]: 2024-03-16 11:10:57.746257044 +0000 UTC m=+0.017385819 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:10:58.139 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:58 smithi078 podman[92354]: 2024-03-16 11:10:58.064810843 +0000 UTC m=+0.335939627 container init d1295b611059406d3e8b14a4959f4dbb4a4facb28c2bd75393dd436c41875e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-16T11:10:58.139 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:58 smithi078 podman[92354]: 2024-03-16 11:10:58.067823341 +0000 UTC m=+0.338952123 container start d1295b611059406d3e8b14a4959f4dbb4a4facb28c2bd75393dd436c41875e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T11:10:58.140 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:58 smithi078 podman[92354]: 2024-03-16 11:10:58.139636354 +0000 UTC m=+0.410765159 container attach d1295b611059406d3e8b14a4959f4dbb4a4facb28c2bd75393dd436c41875e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-16T11:10:58.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:57 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:57.901+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:58.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:57 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:57.901+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:58.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:57 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:57.901+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:58.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:57 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:57.901+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:58.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:57 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:57.901+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:58.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:57 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:57.901+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:10:58.418 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:58 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:58.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:58.418 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:58 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:58.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:58.418 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:58 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:58.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:58.418 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:58 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:58.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:58.418 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:58 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:58.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:58.418 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:58 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:58.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:58.418 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:58 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:58.023+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:10:58.559 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:58 smithi078 podman[92354]: 2024-03-16 11:10:58.353926991 +0000 UTC m=+0.625055769 container died d1295b611059406d3e8b14a4959f4dbb4a4facb28c2bd75393dd436c41875e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T11:10:58.914 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 16 11:10:58 smithi078 podman[92380]: 2024-03-16 11:10:58.893524792 +0000 UTC m=+0.534075466 container remove d1295b611059406d3e8b14a4959f4dbb4a4facb28c2bd75393dd436c41875e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-16T11:10:58.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.9.service' 2024-03-16T11:10:58.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:10:58.961 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-16T11:10:58.961 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-16T11:10:58.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.10 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 systemd[1]: Stopping Ceph osd.10 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.054+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.054+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.054+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.054+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.054+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.055+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.277+0000 7f9af8218700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.277+0000 7f9af8218700 -1 osd.10 93 *** Got signal Terminated *** 2024-03-16T11:10:59.279 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:10:59 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:10:59.277+0000 7f9af8218700 -1 osd.10 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:10:59.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:58.900+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:10:59.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:58.900+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:10:59.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:58.900+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:10:59.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:58.900+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:10:59.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:58.900+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:10:59.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:58.900+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:00.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:59 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:59.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:00.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:59 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:59.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:00.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:59 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:59.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:00.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:59 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:59.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:00.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:59 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:59.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:00.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:10:59 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:10:59.874+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:00.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:00 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:00.059+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:11:00.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:00 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:00.059+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:11:00.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:00 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:00.059+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:11:00.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:00 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:00.059+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:11:00.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:00 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:00.059+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:11:00.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:00 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:00.059+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:11:00.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:00 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:00.059+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:11:01.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:00 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:00.846+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:01.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:00 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:00.846+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:01.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:00 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:00.846+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:01.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:00 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:00.846+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:01.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:00 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:00.846+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:01.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:00 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:00.846+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:01.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:01.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:11:01.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:01.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:11:01.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:01.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:11:01.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:01.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:11:01.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:01.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:11:01.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:01.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:11:01.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:01 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:01.016+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:11:02.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:01 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:01.802+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:02.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:01 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:01.802+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:02.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:01 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:01.802+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:02.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:01 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:01.802+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:02.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:01 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:01.802+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:02.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:01 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:01.802+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:02.294 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:01 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:01.802+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:02.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:02 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:02.014+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:11:02.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:02 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:02.014+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:11:02.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:02 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:02.014+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:11:02.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:02 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:02.014+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:11:02.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:02 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:02.014+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:11:02.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:02 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:02.014+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:11:02.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:02 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:02.014+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:11:03.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:02 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:02.771+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:03.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:02 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:02.771+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:03.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:02 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:02.771+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:03.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:02 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:02.771+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:03.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:02 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:02.771+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:03.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:02 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:02.771+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:03.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:02 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:02.771+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:03.464 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:03 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:03.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:11:03.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:03 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:03.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:11:03.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:03 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:03.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:11:03.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:03 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:03.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:11:03.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:03 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:03.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:11:03.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:03 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:03.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:11:03.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:03 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:03.026+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:11:04.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:03 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:03.733+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:04.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:03 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:03.733+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:04.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:03 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:03.733+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:04.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:03 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:03.733+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:04.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:03 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:03.733+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:04.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:03 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:03.733+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:04.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:03 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:03.733+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:04.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:04.076+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:40.902414+0000 front 2024-03-16T11:09:40.902184+0000 (oldest deadline 2024-03-16T11:10:06.202343+0000) 2024-03-16T11:11:04.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:04.076+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.603599+0000 front 2024-03-16T11:09:52.603573+0000 (oldest deadline 2024-03-16T11:10:15.503717+0000) 2024-03-16T11:11:04.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:04.076+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.305199+0000 front 2024-03-16T11:10:01.305138+0000 (oldest deadline 2024-03-16T11:10:24.204976+0000) 2024-03-16T11:11:04.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:04.076+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-16T11:10:07.606639+0000 front 2024-03-16T11:10:07.606667+0000 (oldest deadline 2024-03-16T11:10:30.506611+0000) 2024-03-16T11:11:04.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:04.076+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:13.907877+0000 front 2024-03-16T11:10:13.907805+0000 (oldest deadline 2024-03-16T11:10:39.807893+0000) 2024-03-16T11:11:04.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:04.076+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:24.810397+0000 front 2024-03-16T11:10:24.810379+0000 (oldest deadline 2024-03-16T11:10:46.510253+0000) 2024-03-16T11:11:04.465 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10[61349]: 2024-03-16T11:11:04.076+0000 7f9af0804700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:32.911875+0000 front 2024-03-16T11:10:32.911837+0000 (oldest deadline 2024-03-16T11:10:53.411766+0000) 2024-03-16T11:11:04.995 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:04 smithi098 podman[70949]: 2024-03-16 11:11:04.716641192 +0000 UTC m=+5.562588434 container died 513be66d1c08f8a7cf9e344c08d8a4be5feb13315a485499a4f23f3f57686ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-16T11:11:05.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:04 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:04.716+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:05.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:04 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:04.716+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:05.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:04 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:04.716+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:05.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:04 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:04.716+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:05.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:04 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:04.716+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:05.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:04 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:04.716+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:05.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:04 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:04.716+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:05.329 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 podman[70949]: 2024-03-16 11:11:05.086902607 +0000 UTC m=+5.932849857 container cleanup 513be66d1c08f8a7cf9e344c08d8a4be5feb13315a485499a4f23f3f57686ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-16T11:11:05.329 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 bash[70949]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10 2024-03-16T11:11:05.639 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 podman[70966]: 2024-03-16 11:11:05.329576473 +0000 UTC m=+0.608931465 container remove 513be66d1c08f8a7cf9e344c08d8a4be5feb13315a485499a4f23f3f57686ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:11:05.933 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 podman[71057]: 2024-03-16 11:11:05.54221553 +0000 UTC m=+0.019211871 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:11:05.933 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 podman[71057]: 2024-03-16 11:11:05.647244374 +0000 UTC m=+0.124240745 container create a434c7bc40e6339bb603cd0cb7d13be9bf3308d096f79c18c7d7ee0d5bcfc80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T11:11:05.933 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 podman[71057]: 2024-03-16 11:11:05.858533101 +0000 UTC m=+0.335529446 container init a434c7bc40e6339bb603cd0cb7d13be9bf3308d096f79c18c7d7ee0d5bcfc80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-16T11:11:05.933 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 podman[71057]: 2024-03-16 11:11:05.861806172 +0000 UTC m=+0.338802502 container start a434c7bc40e6339bb603cd0cb7d13be9bf3308d096f79c18c7d7ee0d5bcfc80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:11:05.933 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:05 smithi098 podman[71057]: 2024-03-16 11:11:05.932893031 +0000 UTC m=+0.409889373 container attach a434c7bc40e6339bb603cd0cb7d13be9bf3308d096f79c18c7d7ee0d5bcfc80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-16T11:11:06.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:05 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:05.736+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:06.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:05 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:05.736+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:06.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:05 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:05.736+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:06.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:05 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:05.736+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:06.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:05 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:05.736+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:06.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:05 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:05.736+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:06.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:05 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:05.736+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:06.214 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 16 11:11:06 smithi098 podman[71057]: 2024-03-16 11:11:06.147337624 +0000 UTC m=+0.624333960 container died a434c7bc40e6339bb603cd0cb7d13be9bf3308d096f79c18c7d7ee0d5bcfc80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T11:11:06.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.10.service' 2024-03-16T11:11:06.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:06.761 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-16T11:11:06.762 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-16T11:11:06.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.11 2024-03-16T11:11:07.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:06.778+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:07.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:06.778+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:07.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:06.778+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:07.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:06.778+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:07.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:06.778+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:07.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:06.778+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:07.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:06.778+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:07.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:06 smithi153 systemd[1]: Stopping Ceph osd.11 for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:07.543 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.051+0000 7f5b86702700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:07.543 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.051+0000 7f5b86702700 -1 osd.11 93 *** Got signal Terminated *** 2024-03-16T11:11:07.543 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.051+0000 7f5b86702700 -1 osd.11 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T11:11:08.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.790+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:08.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.790+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:08.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.790+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:08.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.790+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:08.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.790+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:08.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.790+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:08.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:07 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:07.790+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:09.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:08.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:09.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:08.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:09.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:08.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:09.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:08.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:09.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:08.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:09.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:08.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:09.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:08 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:08.760+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:10.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:09.747+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:10.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:09.747+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:10.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:09.747+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:10.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:09.747+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:10.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:09.747+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:10.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:09.747+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:10.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:09 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:09.747+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:11.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:10.730+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:11.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:10.730+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:11.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:10.730+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:11.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:10.730+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:11.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:10.730+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:11.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:10.730+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:11.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:10 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:10.730+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:12.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:11.724+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-16T11:09:42.594078+0000 front 2024-03-16T11:09:42.593966+0000 (oldest deadline 2024-03-16T11:10:06.694017+0000) 2024-03-16T11:11:12.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:11.724+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-16T11:09:52.595331+0000 front 2024-03-16T11:09:52.595155+0000 (oldest deadline 2024-03-16T11:10:17.895184+0000) 2024-03-16T11:11:12.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:11.724+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-03-16T11:10:01.396113+0000 front 2024-03-16T11:10:01.396007+0000 (oldest deadline 2024-03-16T11:10:23.696149+0000) 2024-03-16T11:11:12.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:11.724+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-16T11:10:16.598581+0000 front 2024-03-16T11:10:16.598394+0000 (oldest deadline 2024-03-16T11:10:41.298322+0000) 2024-03-16T11:11:12.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:11.724+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-03-16T11:10:21.298669+0000 front 2024-03-16T11:10:21.298774+0000 (oldest deadline 2024-03-16T11:10:45.998741+0000) 2024-03-16T11:11:12.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:11.724+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-16T11:10:28.899698+0000 front 2024-03-16T11:10:28.899639+0000 (oldest deadline 2024-03-16T11:10:53.599617+0000) 2024-03-16T11:11:12.044 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:11 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11[61349]: 2024-03-16T11:11:11.724+0000 7f5b82cf6700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-16T11:10:38.101097+0000 front 2024-03-16T11:10:38.101124+0000 (oldest deadline 2024-03-16T11:11:01.601028+0000) 2024-03-16T11:11:12.876 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:12 smithi153 podman[71131]: 2024-03-16 11:11:12.548330434 +0000 UTC m=+5.594435536 container died 557a5085e26bafeb84c1b2fa5d14bb791b5d5109aa9a56e608d21f5a97d9ee78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-16T11:11:12.876 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:12 smithi153 podman[71131]: 2024-03-16 11:11:12.876880889 +0000 UTC m=+5.922986008 container cleanup 557a5085e26bafeb84c1b2fa5d14bb791b5d5109aa9a56e608d21f5a97d9ee78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T11:11:13.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:12 smithi153 bash[71131]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11 2024-03-16T11:11:13.293 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:13 smithi153 podman[71148]: 2024-03-16 11:11:13.036431271 +0000 UTC m=+0.487308948 container remove 557a5085e26bafeb84c1b2fa5d14bb791b5d5109aa9a56e608d21f5a97d9ee78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T11:11:13.714 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:13 smithi153 podman[71240]: 2024-03-16 11:11:13.428868847 +0000 UTC m=+0.110198939 container create c8edcb388aec207334a59fce9f36b466e4e9c3f4693fb81af2da0cedf097c231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:11:13.714 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:13 smithi153 podman[71240]: 2024-03-16 11:11:13.336681528 +0000 UTC m=+0.018011622 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:11:13.714 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:13 smithi153 podman[71240]: 2024-03-16 11:11:13.648162833 +0000 UTC m=+0.329492926 container init c8edcb388aec207334a59fce9f36b466e4e9c3f4693fb81af2da0cedf097c231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:11:13.715 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:13 smithi153 podman[71240]: 2024-03-16 11:11:13.651105911 +0000 UTC m=+0.332436004 container start c8edcb388aec207334a59fce9f36b466e4e9c3f4693fb81af2da0cedf097c231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:11:13.715 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:13 smithi153 podman[71240]: 2024-03-16 11:11:13.714765354 +0000 UTC m=+0.396095469 container attach c8edcb388aec207334a59fce9f36b466e4e9c3f4693fb81af2da0cedf097c231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-03-16T11:11:14.043 INFO:journalctl@ceph.osd.11.smithi153.stdout:Mar 16 11:11:13 smithi153 podman[71240]: 2024-03-16 11:11:13.94497407 +0000 UTC m=+0.626304167 container died c8edcb388aec207334a59fce9f36b466e4e9c3f4693fb81af2da0cedf097c231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T11:11:14.561 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@osd.11.service' 2024-03-16T11:11:14.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:14.602 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-16T11:11:14.602 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-16T11:11:14.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.a 2024-03-16T11:11:14.904 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:11:14 smithi078 systemd[1]: Stopping Ceph mds.a for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:15.309 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:11:14 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a[74596]: 2024-03-16T11:11:14.903+0000 7ff8dbff2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:15.309 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:11:14 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a[74596]: 2024-03-16T11:11:14.903+0000 7ff8dbff2700 -1 mds.a *** got signal Terminated *** 2024-03-16T11:11:16.512 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:11:16 smithi078 podman[92473]: 2024-03-16 11:11:16.195179663 +0000 UTC m=+1.399466190 container died 67e9c56b4cf471beee5310dd0a1c937cc0c446799299680c13fc72fe01915b22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T11:11:16.809 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:11:16 smithi078 podman[92473]: 2024-03-16 11:11:16.595139912 +0000 UTC m=+1.799426438 container cleanup 67e9c56b4cf471beee5310dd0a1c937cc0c446799299680c13fc72fe01915b22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-16T11:11:16.809 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 16 11:11:16 smithi078 bash[92473]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-a 2024-03-16T11:11:16.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.a.service' 2024-03-16T11:11:16.984 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:16.985 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-16T11:11:16.985 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-16T11:11:16.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.d 2024-03-16T11:11:17.298 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:17 smithi078 systemd[1]: Stopping Ceph mds.d for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:17.559 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d[73634]: 2024-03-16T11:11:17.298+0000 7f2b0885c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:17.559 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:17 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d[73634]: 2024-03-16T11:11:17.298+0000 7f2b0885c700 -1 mds.d *** got signal Terminated *** 2024-03-16T11:11:21.357 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:21 smithi078 podman[92594]: 2024-03-16 11:11:21.099148751 +0000 UTC m=+3.895100425 container died 721c557738ef6dedcdf944c6fb4b49e24354d0366a847db51c9530f61a498693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T11:11:21.625 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:21 smithi078 podman[92594]: 2024-03-16 11:11:21.440667615 +0000 UTC m=+4.236619332 container cleanup 721c557738ef6dedcdf944c6fb4b49e24354d0366a847db51c9530f61a498693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-03-16T11:11:21.625 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:21 smithi078 bash[92594]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d 2024-03-16T11:11:21.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.d.service' 2024-03-16T11:11:21.878 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:21 smithi078 podman[92608]: 2024-03-16 11:11:21.625217659 +0000 UTC m=+0.528893869 container remove 721c557738ef6dedcdf944c6fb4b49e24354d0366a847db51c9530f61a498693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-16T11:11:21.878 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:21 smithi078 systemd[1]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.d.service: Deactivated successfully. 2024-03-16T11:11:21.878 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 16 11:11:21 smithi078 systemd[1]: Stopped Ceph mds.d for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:11:21.890 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:21.890 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-16T11:11:21.890 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-16T11:11:21.890 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.g 2024-03-16T11:11:22.210 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 16 11:11:21 smithi078 systemd[1]: Stopping Ceph mds.g for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:22.559 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 16 11:11:22 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-g[74236]: 2024-03-16T11:11:22.209+0000 7fef21587700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:22.559 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 16 11:11:22 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-g[74236]: 2024-03-16T11:11:22.209+0000 7fef21587700 -1 mds.g *** got signal Terminated *** 2024-03-16T11:11:25.803 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 16 11:11:25 smithi078 podman[92714]: 2024-03-16 11:11:25.448843461 +0000 UTC m=+3.344767627 container died 6a18916af149b84d1ead8abdcc51a01f00b6e7bb84e0e173ff4f036213cecf70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T11:11:26.059 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 16 11:11:25 smithi078 podman[92714]: 2024-03-16 11:11:25.894136148 +0000 UTC m=+3.790060313 container cleanup 6a18916af149b84d1ead8abdcc51a01f00b6e7bb84e0e173ff4f036213cecf70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:11:26.059 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 16 11:11:25 smithi078 bash[92714]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-g 2024-03-16T11:11:26.321 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.g.service' 2024-03-16T11:11:26.355 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:26.355 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-16T11:11:26.355 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-16T11:11:26.356 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.j 2024-03-16T11:11:26.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:26 smithi078 systemd[1]: Stopping Ceph mds.j for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:27.059 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j[73246]: 2024-03-16T11:11:26.688+0000 7f67afaa2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:27.059 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:26 smithi078 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j[73246]: 2024-03-16T11:11:26.688+0000 7f67afaa2700 -1 mds.j *** got signal Terminated *** 2024-03-16T11:11:30.559 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:30 smithi078 podman[92835]: 2024-03-16 11:11:30.295542477 +0000 UTC m=+3.727735003 container died e69bad745878ee0910592101bbfa330aa4bcf9c96367f3bd654a629113874f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-16T11:11:30.915 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:30 smithi078 podman[92835]: 2024-03-16 11:11:30.739531577 +0000 UTC m=+4.171724104 container cleanup e69bad745878ee0910592101bbfa330aa4bcf9c96367f3bd654a629113874f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:11:30.915 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:30 smithi078 bash[92835]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j 2024-03-16T11:11:31.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.j.service' 2024-03-16T11:11:31.180 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:30 smithi078 podman[92850]: 2024-03-16 11:11:30.915547479 +0000 UTC m=+0.617663116 container remove e69bad745878ee0910592101bbfa330aa4bcf9c96367f3bd654a629113874f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-j, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T11:11:31.180 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:31 smithi078 systemd[1]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.j.service: Deactivated successfully. 2024-03-16T11:11:31.181 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 16 11:11:31 smithi078 systemd[1]: Stopped Ceph mds.j for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:11:31.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:31.191 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-16T11:11:31.191 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-16T11:11:31.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.b 2024-03-16T11:11:31.488 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:11:31 smithi098 systemd[1]: Stopping Ceph mds.b for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:31.739 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:11:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: 2024-03-16T11:11:31.486+0000 7f5133fa5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:31.739 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:11:31 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b[67019]: 2024-03-16T11:11:31.486+0000 7f5133fa5700 -1 mds.b *** got signal Terminated *** 2024-03-16T11:11:33.689 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:11:33 smithi098 podman[71164]: 2024-03-16 11:11:33.424919118 +0000 UTC m=+2.028720592 container died 90329a53074356e660e6c17e941564510f7084da9a27e960e4e48b82e71ebac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-16T11:11:33.964 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:11:33 smithi098 podman[71164]: 2024-03-16 11:11:33.779100544 +0000 UTC m=+2.382902031 container cleanup 90329a53074356e660e6c17e941564510f7084da9a27e960e4e48b82e71ebac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:11:33.964 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 16 11:11:33 smithi098 bash[71164]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-b 2024-03-16T11:11:34.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.b.service' 2024-03-16T11:11:34.276 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:34.277 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-16T11:11:34.277 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-16T11:11:34.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.e 2024-03-16T11:11:34.609 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:34 smithi098 systemd[1]: Stopping Ceph mds.e for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:34.964 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e[66385]: 2024-03-16T11:11:34.608+0000 7f735fda9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:34.964 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:34 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e[66385]: 2024-03-16T11:11:34.608+0000 7f735fda9700 -1 mds.e *** got signal Terminated *** 2024-03-16T11:11:37.693 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:37 smithi098 podman[71296]: 2024-03-16 11:11:37.426689076 +0000 UTC m=+2.934578655 container died 84f36ef85f500b1579fc9dd8f56e701585ff618a7763262cb7cb7f16d68a564f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-16T11:11:37.951 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:37 smithi098 podman[71296]: 2024-03-16 11:11:37.767124925 +0000 UTC m=+3.275014547 container cleanup 84f36ef85f500b1579fc9dd8f56e701585ff618a7763262cb7cb7f16d68a564f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:11:37.951 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:37 smithi098 bash[71296]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e 2024-03-16T11:11:38.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.e.service' 2024-03-16T11:11:38.214 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:37 smithi098 podman[71310]: 2024-03-16 11:11:37.951538109 +0000 UTC m=+0.518587253 container remove 84f36ef85f500b1579fc9dd8f56e701585ff618a7763262cb7cb7f16d68a564f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-03-16T11:11:38.214 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:38 smithi098 systemd[1]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.e.service: Deactivated successfully. 2024-03-16T11:11:38.214 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:38 smithi098 systemd[1]: Stopped Ceph mds.e for a1941e5e-e383-11ee-95c9-87774f69a715. 2024-03-16T11:11:38.214 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 16 11:11:38 smithi098 systemd[1]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.e.service: Consumed 1.273s CPU time. 2024-03-16T11:11:38.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:38.228 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-16T11:11:38.228 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-16T11:11:38.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.h 2024-03-16T11:11:38.537 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:11:38 smithi098 systemd[1]: Stopping Ceph mds.h for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:38.964 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:11:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h[66696]: 2024-03-16T11:11:38.536+0000 7ff6ce47b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:38.964 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:11:38 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h[66696]: 2024-03-16T11:11:38.536+0000 7ff6ce47b700 -1 mds.h *** got signal Terminated *** 2024-03-16T11:11:41.714 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:11:41 smithi098 podman[71419]: 2024-03-16 11:11:41.423974942 +0000 UTC m=+2.981980477 container died 3fcbdba0c316768b1112d792005fe04394c94d1290eec7870722ab7c24f70a2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-16T11:11:42.004 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:11:41 smithi098 podman[71419]: 2024-03-16 11:11:41.729333284 +0000 UTC m=+3.287338819 container cleanup 3fcbdba0c316768b1112d792005fe04394c94d1290eec7870722ab7c24f70a2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-16T11:11:42.004 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:11:41 smithi098 bash[71419]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h 2024-03-16T11:11:42.004 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 16 11:11:41 smithi098 podman[71433]: 2024-03-16 11:11:41.96404381 +0000 UTC m=+0.531557049 container remove 3fcbdba0c316768b1112d792005fe04394c94d1290eec7870722ab7c24f70a2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T11:11:42.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.h.service' 2024-03-16T11:11:42.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:42.126 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-16T11:11:42.126 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-16T11:11:42.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.k 2024-03-16T11:11:42.441 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:11:42 smithi098 systemd[1]: Stopping Ceph mds.k for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:42.714 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:11:42 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k[66063]: 2024-03-16T11:11:42.440+0000 7f78b9f39700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:42.715 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:11:42 smithi098 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k[66063]: 2024-03-16T11:11:42.440+0000 7f78b9f39700 -1 mds.k *** got signal Terminated *** 2024-03-16T11:11:43.833 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:11:43 smithi098 podman[71539]: 2024-03-16 11:11:43.51603014 +0000 UTC m=+1.175007844 container died c3d1d4e297309b26ff87396a7aca659fee0cf9dd81dd46d19962f57941fe61c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True) 2024-03-16T11:11:44.214 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:11:43 smithi098 podman[71539]: 2024-03-16 11:11:43.900008194 +0000 UTC m=+1.558985897 container cleanup c3d1d4e297309b26ff87396a7aca659fee0cf9dd81dd46d19962f57941fe61c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-16T11:11:44.214 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 16 11:11:43 smithi098 bash[71539]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-k 2024-03-16T11:11:44.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.k.service' 2024-03-16T11:11:44.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:44.416 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-16T11:11:44.416 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-16T11:11:44.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.c 2024-03-16T11:11:44.751 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:11:44 smithi153 systemd[1]: Stopping Ceph mds.c for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:45.043 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:11:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c[66789]: 2024-03-16T11:11:44.751+0000 7f94db9a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:45.043 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:11:44 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c[66789]: 2024-03-16T11:11:44.751+0000 7f94db9a6700 -1 mds.c *** got signal Terminated *** 2024-03-16T11:11:49.043 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:11:48 smithi153 podman[71345]: 2024-03-16 11:11:48.760064328 +0000 UTC m=+4.148067179 container died 43af1a72d8d1fbacc50525a72ab2ca50f1bdb026f32afa00095fd3904a1eae13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-16T11:11:49.359 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:11:49 smithi153 podman[71345]: 2024-03-16 11:11:49.117181478 +0000 UTC m=+4.505184338 container cleanup 43af1a72d8d1fbacc50525a72ab2ca50f1bdb026f32afa00095fd3904a1eae13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T11:11:49.360 INFO:journalctl@ceph.mds.c.smithi153.stdout:Mar 16 11:11:49 smithi153 bash[71345]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-c 2024-03-16T11:11:49.497 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.c.service' 2024-03-16T11:11:49.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:49.534 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-16T11:11:49.535 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-16T11:11:49.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.f 2024-03-16T11:11:50.043 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:11:49 smithi153 systemd[1]: Stopping Ceph mds.f for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:50.043 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:11:49 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f[66481]: 2024-03-16T11:11:49.829+0000 7f1d4da14700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:50.043 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:11:49 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f[66481]: 2024-03-16T11:11:49.829+0000 7f1d4da14700 -1 mds.f *** got signal Terminated *** 2024-03-16T11:11:53.757 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:11:53 smithi153 podman[71478]: 2024-03-16 11:11:53.427296719 +0000 UTC m=+3.680455813 container died 32a197345a73294f2ddfe22bf72922244e66154febcd53aabb3b34f54aab2285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:11:54.043 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:11:53 smithi153 podman[71478]: 2024-03-16 11:11:53.837097861 +0000 UTC m=+4.090256962 container cleanup 32a197345a73294f2ddfe22bf72922244e66154febcd53aabb3b34f54aab2285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-16T11:11:54.043 INFO:journalctl@ceph.mds.f.smithi153.stdout:Mar 16 11:11:53 smithi153 bash[71478]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-f 2024-03-16T11:11:54.211 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.f.service' 2024-03-16T11:11:54.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:54.247 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-16T11:11:54.247 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-16T11:11:54.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.i 2024-03-16T11:11:54.543 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:11:54 smithi153 systemd[1]: Stopping Ceph mds.i for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:55.044 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:11:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i[67102]: 2024-03-16T11:11:54.606+0000 7fc7bed59700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:55.044 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:11:54 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i[67102]: 2024-03-16T11:11:54.606+0000 7fc7bed59700 -1 mds.i *** got signal Terminated *** 2024-03-16T11:11:57.692 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:11:57 smithi153 podman[71599]: 2024-03-16 11:11:57.432905095 +0000 UTC m=+2.970073625 container died 2c6276463a8100e42a223000648676126d27aa0a2e6d60c976872fdf0833b255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:11:58.043 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:11:57 smithi153 podman[71599]: 2024-03-16 11:11:57.782924685 +0000 UTC m=+3.320093213 container cleanup 2c6276463a8100e42a223000648676126d27aa0a2e6d60c976872fdf0833b255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T11:11:58.043 INFO:journalctl@ceph.mds.i.smithi153.stdout:Mar 16 11:11:57 smithi153 bash[71599]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-i 2024-03-16T11:11:58.195 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.i.service' 2024-03-16T11:11:58.230 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:11:58.230 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-16T11:11:58.231 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-16T11:11:58.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.l 2024-03-16T11:11:58.793 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:11:58 smithi153 systemd[1]: Stopping Ceph mds.l for a1941e5e-e383-11ee-95c9-87774f69a715... 2024-03-16T11:11:58.793 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:11:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l[66168]: 2024-03-16T11:11:58.560+0000 7f15b7ff0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:11:58.793 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:11:58 smithi153 ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l[66168]: 2024-03-16T11:11:58.560+0000 7f15b7ff0700 -1 mds.l *** got signal Terminated *** 2024-03-16T11:12:00.986 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:12:00 smithi153 podman[71719]: 2024-03-16 11:12:00.713761359 +0000 UTC m=+2.258603556 container died 9455dd4412f1f247277bdcc292671e22dd8b57725e1d3da708a8759434777c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T11:12:01.293 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:12:01 smithi153 podman[71719]: 2024-03-16 11:12:01.086589475 +0000 UTC m=+2.631431658 container cleanup 9455dd4412f1f247277bdcc292671e22dd8b57725e1d3da708a8759434777c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-16T11:12:01.293 INFO:journalctl@ceph.mds.l.smithi153.stdout:Mar 16 11:12:01 smithi153 bash[71719]: ceph-a1941e5e-e383-11ee-95c9-87774f69a715-mds-l 2024-03-16T11:12:01.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1941e5e-e383-11ee-95c9-87774f69a715@mds.l.service' 2024-03-16T11:12:01.548 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T11:12:01.548 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-16T11:12:01.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1941e5e-e383-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T11:12:01.746 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T11:12:04.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1941e5e-e383-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T11:12:04.321 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T11:12:06.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1941e5e-e383-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T11:12:06.859 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T11:12:09.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:12:09.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:12:09.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:12:09.256 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-16T11:12:09.258 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi078/crash 2024-03-16T11:12:09.259 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash -- . 2024-03-16T11:12:09.302 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T11:12:09.302 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2024-03-16T11:12:09.304 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi098/crash 2024-03-16T11:12:09.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash -- . 2024-03-16T11:12:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T11:12:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-03-16T11:12:09.355 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi153/crash 2024-03-16T11:12:09.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash -- . 2024-03-16T11:12:09.404 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T11:12:09.404 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-03-16T11:12:09.405 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-16T11:12:09.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T11:12:09.443 INFO:teuthology.orchestra.run.smithi078.stdout:2024-03-16T10:58:01.243454+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T11:12:09.444 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-16T11:12:09.444 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[SEC\]' /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T11:12:09.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]' /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T11:12:09.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[WRN\]' /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T11:12:09.666 INFO:teuthology.orchestra.run.smithi078.stdout:2024-03-16T10:58:01.243454+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T11:12:09.666 INFO:tasks.cephadm:Compressing logs... 2024-03-16T11:12:09.667 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T11:12:09.709 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T11:12:09.712 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T11:12:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T11:12:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:: No such file or directory 2024-03-16T11:12:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-16T11:12:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-16T11:12:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T11:12:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-16T11:12:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-16T11:12:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-16T11:12:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-16T11:12:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-16T11:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T11:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-03-16T11:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-16T11:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T11:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-16T11:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-16T11:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.x.log: 90.2% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:gzip 94.6% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.0.log: 94.7% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log: 83.0% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log: 86.3% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T11:12:09.745 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-16T11:12:09.746 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-16T11:12:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.10.log: /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.b.log: -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T11:12:09.751 INFO:teuthology.orchestra.run.smithi098.stderr: 94.8% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-16T11:12:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.d.log: 94.7% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-16T11:12:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-16T11:12:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.i.log: /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.11.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.f.log: 94.8% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr: 90.4% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-16T11:12:09.755 INFO:teuthology.orchestra.run.smithi153.stderr: 82.6% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T11:12:09.756 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log: 86.2% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T11:12:09.756 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- 92.2% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-16T11:12:09.756 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-16T11:12:09.756 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T11:12:09.756 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-16T11:12:09.756 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.0.log: 91.0% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T11:12:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-16T11:12:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T11:12:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.y.log: 92.0% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-16T11:12:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log 2024-03-16T11:12:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T11:12:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T11:12:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log: 86.2% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T11:12:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-16T11:12:09.774 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T11:12:09.780 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-16T11:12:09.780 INFO:teuthology.orchestra.run.smithi153.stderr: 92.6% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-16T11:12:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log: 92.6% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-16T11:12:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T11:12:09.802 INFO:teuthology.orchestra.run.smithi098.stderr: 94.0% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-16T11:12:09.809 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-16T11:12:09.930 INFO:teuthology.orchestra.run.smithi078.stderr: 89.7% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-16T11:12:09.949 INFO:teuthology.orchestra.run.smithi098.stderr: 92.7% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-16T11:12:09.962 INFO:teuthology.orchestra.run.smithi153.stderr: 91.8% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-16T11:12:10.042 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-16T11:12:10.267 INFO:teuthology.orchestra.run.smithi078.stderr: 90.6% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-16T11:12:11.601 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-16T11:12:12.174 INFO:teuthology.orchestra.run.smithi078.stderr: 93.5% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-16T11:12:12.245 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-16T11:12:12.544 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-16T11:12:12.564 INFO:teuthology.orchestra.run.smithi078.stderr: 93.7% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-16T11:12:12.673 INFO:teuthology.orchestra.run.smithi078.stderr: 93.4% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-16T11:12:13.029 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-16T11:12:13.047 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-16T11:12:13.362 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-16T11:12:13.364 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-16T11:12:13.364 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m3.638s 2024-03-16T11:12:13.364 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m12.328s 2024-03-16T11:12:13.365 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.492s 2024-03-16T11:12:13.536 INFO:teuthology.orchestra.run.smithi153.stderr: 93.8% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-16T11:12:13.706 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-16T11:12:13.707 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-16T11:12:13.707 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m3.981s 2024-03-16T11:12:13.707 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m12.790s 2024-03-16T11:12:13.707 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.496s 2024-03-16T11:12:16.204 INFO:teuthology.orchestra.run.smithi078.stderr: 93.5% -- replaced with /var/log/ceph/a1941e5e-e383-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-16T11:12:19.346 INFO:teuthology.orchestra.run.smithi078.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-16T11:12:19.348 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-03-16T11:12:19.348 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m9.627s 2024-03-16T11:12:19.348 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m24.115s 2024-03-16T11:12:19.348 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.897s 2024-03-16T11:12:19.349 INFO:tasks.cephadm:Archiving logs... 2024-03-16T11:12:19.349 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi078/log 2024-03-16T11:12:19.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T11:12:22.313 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi098/log 2024-03-16T11:12:22.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T11:12:24.176 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi153/log 2024-03-16T11:12:24.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T11:12:26.138 INFO:tasks.cephadm:Removing cluster... 2024-03-16T11:12:26.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1941e5e-e383-11ee-95c9-87774f69a715 --force 2024-03-16T11:12:26.330 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T11:12:26.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1941e5e-e383-11ee-95c9-87774f69a715 --force 2024-03-16T11:12:26.795 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T11:12:27.211 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1941e5e-e383-11ee-95c9-87774f69a715 --force 2024-03-16T11:12:27.404 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: a1941e5e-e383-11ee-95c9-87774f69a715 2024-03-16T11:12:27.681 INFO:tasks.cephadm:Removing cephadm ... 2024-03-16T11:12:27.682 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T11:12:27.700 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T11:12:27.716 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T11:12:27.731 INFO:tasks.cephadm:Teardown complete 2024-03-16T11:12:27.731 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-16T11:12:27.743 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-16T11:12:27.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T11:12:27.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T11:12:27.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T11:12:27.870 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T11:12:27.882 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T11:12:27.882 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T11:12:27.885 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T11:12:27.887 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T11:12:27.898 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-03-16T11:12:27.899 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-16T11:12:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-16T11:12:27.904 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T11:12:27.904 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-16T11:12:27.904 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +699us[ +689us] +/- 92ms 2024-03-16T11:12:27.904 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -518us[ -528us] +/- 60ms 2024-03-16T11:12:27.904 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +95us[ +85us] +/- 68ms 2024-03-16T11:12:27.905 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:12:27.905 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T11:12:27.905 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-03-16T11:12:27.905 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +714us[ +695us] +/- 92ms 2024-03-16T11:12:27.905 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -530us[ -549us] +/- 60ms 2024-03-16T11:12:27.905 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +83us[ +64us] +/- 68ms 2024-03-16T11:12:27.905 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:12:27.906 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T11:12:27.906 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-16T11:12:27.906 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +691us[ +674us] +/- 92ms 2024-03-16T11:12:27.906 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -543us[ -561us] +/- 60ms 2024-03-16T11:12:27.906 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +66us[ +66us] +/- 68ms 2024-03-16T11:12:27.906 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:12:27.907 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T11:12:27.917 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T11:12:27.918 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T11:12:27.928 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T11:12:28.029 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T11:12:28.098 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T11:12:28.168 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T11:12:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:12:28.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T11:12:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:12:28.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T11:12:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:12:28.259 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T11:12:28.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T11:12:28.280 INFO:teuthology.task.internal:Duration was 1408.170587 seconds 2024-03-16T11:12:28.280 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T11:12:28.290 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T11:12:28.290 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T11:12:28.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T11:12:28.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T11:12:28.334 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T11:12:28.335 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T11:12:28.342 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T11:12:28.756 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T11:12:28.756 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-03-16T11:12:28.756 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T11:12:28.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T11:12:28.778 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T11:12:28.821 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-16T11:12:28.822 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T11:12:28.849 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T11:12:28.849 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T11:12:28.851 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T11:12:28.864 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T11:12:28.918 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T11:12:28.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T11:12:28.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T11:12:28.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T11:12:29.275 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T11:12:29.286 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T11:12:29.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T11:12:29.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T11:12:29.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T11:12:29.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T11:12:29.381 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T11:12:29.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T11:12:29.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T11:12:29.406 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-03-16T11:12:29.409 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-16T11:12:29.435 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-16T11:12:29.461 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T11:12:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:12:29.476 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T11:12:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:12:29.491 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T11:12:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:12:29.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T11:12:29.530 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T11:12:29.531 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi078 2024-03-16T11:12:29.531 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T11:12:29.575 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi098 2024-03-16T11:12:29.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T11:12:29.611 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383/remote/smithi153 2024-03-16T11:12:29.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T11:12:29.650 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T11:12:29.650 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T11:12:29.653 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T11:12:29.655 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T11:12:29.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T11:12:29.713 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T11:12:29.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T11:12:29.744 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T11:12:29.745 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T11:12:29.747 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T11:12:29.749 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T11:12:29.759 INFO:teuthology.orchestra.run.smithi078.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 11:12 /home/ubuntu/cephtest 2024-03-16T11:12:29.761 INFO:teuthology.orchestra.run.smithi098.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 11:12 /home/ubuntu/cephtest 2024-03-16T11:12:29.764 INFO:teuthology.orchestra.run.smithi153.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 11:12 /home/ubuntu/cephtest 2024-03-16T11:12:29.765 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T11:12:29.776 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T11:12:29.801 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T11:12:29.828 DEBUG:teuthology.nuke:shortname: smithi078 2024-03-16T11:12:29.829 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T11:12:29.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 10:36:34.462209', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLnzJV8ne5gyMIf4urpZbqqOlFYqmyuQrZ84m/vYtwhJEPOOCnWA0r4FQLlgUfWqimFrPtiEAvgk8oTorFKu604='} 2024-03-16T11:12:29.857 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-16T11:12:29.858 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T11:12:29.883 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 10:36:34.458931', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCada2N07IWP7+LB714V7XpZGmIcV/4JdlYfwZDuV2LVJZwQnEpnN6kIlHzodmn8slSz0nSuW9XfkIz2zd+scz0='} 2024-03-16T11:12:29.884 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-16T11:12:29.884 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T11:12:29.905 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602383', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 10:36:34.460556', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLa/MvBntgRDRa3uM9RWScCoK7JGGM2OEGQQFG6Q3S8KBqu94BYRpJjQp5T9A3+8mevxB8ToVb7veo2yhthHagY='} 2024-03-16T11:12:29.914 INFO:teuthology.orchestra.console.smithi078:Power off 2024-03-16T11:12:29.915 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T11:12:29.942 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-03-16T11:12:29.942 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:29.961 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-16T11:12:29.961 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T11:12:29.971 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-03-16T11:12:29.982 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-16T11:12:29.982 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:29.995 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-16T11:12:29.995 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T11:12:30.005 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-16T11:12:30.015 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-16T11:12:30.016 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:30.039 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-16T11:12:33.972 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:34.006 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:34.039 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:34.082 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-03-16T11:12:34.117 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-16T11:12:34.149 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-16T11:12:38.084 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:38.118 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:38.150 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:38.200 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-03-16T11:12:38.230 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-16T11:12:38.261 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-16T11:12:38.330 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-16T11:12:38.362 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-16T11:12:42.204 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T11:12:42.313 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-03-16T11:12:42.414 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-03-16T11:12:42.518 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} duration: 1408.1705868244171 failure_reason: '"2024-03-16T10:58:01.243454+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-16T11:12:42.519 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T11:12:42.601 INFO:teuthology.run:FAIL